utTfoxshnical  note  techn 


AD-A269  473 


Influence  of  Individual 
Experience  and  Flight  Strips 
on  Air  ^rtths^jatroller 
Mentdry  /  Situatio! 


This  document  is  avaJIaBTSl^lhe  public 
through  the  National  Technical  Information 
Service,  Springfield,  Virginia  22161 


© 

US  Deporimer*  of  lansponaHon 


Ttctmtetl  Cwiif 

Alkmto  CWy  hrtfnrtton*  Airport.  N.J.  08408 


93 


NOTICE 

This  document  is  disseminated  under  the  sponsorship 
of  the  U.S.  Department  of  Transportation  In  the  interest  of 
information  exchange.  The  United  States  Government 
assumes  no  liabflity  tor  the  contents  or  use  thereof. 

The  United  States  Government  does  not  endorse 
products  or  manufacturers.  Trade  or  manufacturers’ 
names  appear  herein  solely  because  they  are  considered 
essential  to  the  objective  of  this  report 


1.  Rtpprf  No. 


2.  Govtmmtni  Acctttton  No. 


Technical  Report  Documentation  Pag* 

3.  Recipient' •  Cateiog  No. 


DOT/FAA/CT-TN9  3/31 


lnfiuenceuof,*Individual  Experience  and  Flight  Strips 
on  Air  Traffic  Controller  Memory/Situational  Awareness 


Carolina  Zingale,  PhD,  Stan  Gromelski,  S.  Basheer 


9.  Performing  Organisation  Nome  and  Addros* 

PERI  (Princeton  Economic  Research,  Inc.) 

322  Wall  Street 

Princeton,  New  Jersey  08540 


12.  Sponsoring  Agency  Nome  ond  Addreee 

Federal  Aviation  Administration 
Technical  Center 

Engineering,  Research,  and  Development  Service 
Atlantic  City  International  Airport,  NJ  08405 


15.  Supplementary  Notes 


5.  Report  Oote 

June,  1993 


5.  Performing  Organisation  Code 


8.  Performing  Orgoni lotion  Report  No. 

DOT/FAA/CT-TN93/31 


10.  Work  Unit  No.  (TRAIS) 


11.  Contract  or  Grant  No. 

DTFA03-89-00050 


13.  Typo  of  Report  end  Period  Covered 

Technical  Note 

June  1992  -  February  1992 


14.  Sponsoring  Agency  Code 

ACD-  350 


14.  Abstroet 

An  experiment  was  conducted  to  compare  situational  awareness  and  control  ability  between  two  test  conditions.  One 
condition  allowed  for  the  active  organization  of  aircraft  information  (allowed  note-writing  on  flight  strips  containing 
directional  arrows).  The  other  did  not  (note-writing  not  allowed,  no  arrows  on  strips).  Such  flight  strip  management 
activities  have  been  cited  as  critical  for  the  formation  and  maintenance  of  the  controllers’  ’picture." 

Participants  were  eight  air  traffic  controllers  with  at  least  four  years  of  on-the-job  experience.  They  were  tested  using 
TRACON  n,  an  air  traffic  control  (ATC)  simulator  for  the  personal  computer  (pc)  which  requires  keyboard-entered, 
rather  than  verbally-issued  commands.  Situational  awareness  was  evaluated  during  intervals  in  which  each  of  the  test 
scenarios  was  paused  and  the  display  was  blocked  from  view.  Participants  indicated  the  locations  of  all  active  aircraft 
on  a  paper  map  and  reported  the  last  command  issued  to  each.  Control  performance  was  evaluated  using  a  variation  of 
the  TRACON-provided  scoring  system. 

Situational  awareness  and  control  performance  did  not  differ  as  a  function  of  test  condition.  However,  situational 
awareness  was  found  to  differ  as  a  function  of  reported  level  of  video-game  experience.  Controllers  who  reported  lower 
levels  of  video-game  experience  on  a  pre-test  questionnaire,  showed  improved  situational  awareness  when  they  were 
allowed  to  write  on  the  strips.  Those  who  reported  higher  video-game  experience  maintained  higher  situational  awareness 
across  both  conditions.  Further  analyses  indicated  that  these  groups  differed  in  terms  of  their  memory  for  last-issued 
commands  (and  more  specifically,  for  last-assigned  altitudes),  not  their  memory  for  aircraft  locations. 

Regardless  of  video-game  experience,  controllers  reported  that  their  ability  to  remember  call  signs  in  this  experiment  was 
lower  than  it  is  on  the  job.  Having  to  issue  commands  via  the  keyboard,  instead  of  verbally,  may  have  produced  this 
result.  The  implications  of  these  results  for  systems  requiring  such  interaction  are  discussed. 


17.  Key  Word* 

Air  Traffic  Control 
Controller  Memory 


Flight  Strips 
Note-writing 


18.  OiitrikutiM  Statement 

Available  to  the  public  through  the 
National  Technical  Information  Service 


Situational  Awareness 

Memory  Enhancement 

Springfield,  VA 

22161 

19.  Security  Gotti  f.  (of  tfiit  rege ft) 

20.  Security  CI.Mil.  (of  thi»  peg.) 

21.  No.  of  Peg.* 

22.  Price 

Unclassified 

Unclassified 

73 

Form  00T  F  1730., 


Reproduction  of  completed  pop.  authorised 


TABLE  OF  CONTENTS 


Page 


EXECUTIVE  SUMMARY 


vii 


1.  INTRODUCTION  1 

1 . 1  Prologue  1 

1.2  Background  of  ATC  Memory  Research  2 

1.3  Memory  Research  Project  at  the  FAA  Technical  Center  4 

2.  EXPERIMENT:  USEFULNESS  OF  FLIGHT  STRIPS  (NOTE-WRITING 

AND  ORGANIZATION)  TO  SITUATIONAL  AWARENESS  AND 
PERFORMANCE  7 

2.1  Purpose  7 

2.2  Method  7 


2.2.1  Participants  7 

2.2.2  Equipment  8 

2.2.3  Training  ®TIC  Qu/iuu*  iKSrLCTSO  &  9 

2.2.4  Test  Conditions  11 

2.2.5  Performance  Measures  11 

2.2.6  Procedure  12 

2.2.7  Design  14 


3.  RESULTS 

3.1  Additional  Analyses 

3.2  Other  Factors  Affecting  Performance 

4.  DISCUSSION 

5.  RECOMMENDATIONS 

6.  IMPLICATIONS 
REFERENCES 
APPENDIXES 

A  -  Pilot  Studies  with  Non-controllers 
B  -  Training  Materials 
C  -  Flight  Strips  and  Response  Forms 
D  -  Questionnaires  and  Scoring  Sheets 


Accesion  For  j  j 

NTIS 

CRA&I 

or  ic 

I AB 

□ 

Uf  »d.’ii  • 

JiiitifK 

ori*iced 

:ation 

i— 

By 

Distribution/ 

Availability  Codes 

Dist 

Avail  and/or 

Special 

f\-l 

15 

16 
20 

24 

25 

26 
28 


iii 


LIST  OF  ILLUSTRATIONS 


Figure  Page 

3-1  Mean  SA  Score  as  a  Function  of  Condition  and  Test  Order  15 

3-2  SA  Score  as  a  Function  of  Video-Game  Experience  20 

3-3  Mean  Proportion  Correct:  Locations  of  Aircraft  21 

3-4  Mean  Proportion  Correct:  Last  Command  Issued  22 

3-5  Mean  Proportion  Correct:  Last  Altitude  Change  Issued  23 

LIST  OF  TABLES 

Table  Page 

2- 1  Experimental  Design  14 

3- 1  Correlation  Matrix  for  Preliminary  Questionnaire  Variables  and  17 

SA  in  the  No  Writing  Condition 

3-2  Proportion  of  Commands  Written  by  Type  17 


tv 


ACKNOWLEDGMENTS 


We  gratefully  acknowledge  the  participation  of  controllers  from  the  FAA  Technical  Center 
and  the  Atlantic  City  International  Airport,  who  volunteered  their  time  and  effort  to  support 
this  project.  We  would  also  like  to  thank  the  non-controller  participants  who  took  part  in 
pilot  work  on  this  project,  employees  of  the  FAA  Technical  Center  and  students  from  the 
Aviation  Program  at  Mercer  County  Community  College,  West  Windsor,  NJ. 


v 


EXECUTIVE  SUMMARY 


Situational  awareness  and  control  ability  were  measured  under  two  test  conditions  using  a  low- 
fidelity,  personal  computer  (pc)  based  simulator  (TRACON  II).  Unlike  the  actual  air  traffic 
control  (ATC)  environment,  TRACON  II  requires  that  commands  be  issued  via  keyboard,  rather 
than  verbally.  Participants  were  air  traffic  control  specialists  (ATCS)  who  were  either  current 
or  former  controllers  with  experience  in  terminal  environments.  The  purpose  of  the  experiment 
was  to  determine  the  impact  of  note-writing  and  flight  strip  organization  on  situational  awareness 
and  performance.  A  secondary  purpose  of  the  experiment  was  to  determine  the  efficacy  of  using 
this  type  of  simulator  to  conduct  research  with  air  traffic  controllers. 

In  one  test  condition,  participants  kept  notes  on  flight  strips  that  included  arrows  indicating 
whether  aircraft  were  arrivals  (i),  departures  (t),  or  overflights  (**).  In  the  second  condition, 
note-writing  was  not  allowed  and  arrows  were  not  presented.  Situational  awareness  was 
measured  by  stopping  the  scenario,  blocking  it  from  view,  and  requiring  participants  to  record 
the  location  of  each  active  aircraft  on  an  airspace  map  and  to  report  the  last  command  issued  to 
each. 

Participants  also  completed  pre-  and  post-session  questionnaires  designed  to  identify  other  factors 
potentially  relevant  to  performance,  such  as  level  of  ATC  experience,  computer  or  video-game 
experience,  motivation  to  participate,  stress  level,  and  workload  level.  Participants  also 
described  their  control  strategies  and  indicated  anything  that  they  felt  had  helped  or  hurt  their 
performance  in  the  sessions. 

Neither  situational  awareness  nor  control  performance  differed  significantly  between  test 
conditions.  However,  situational  awareness  did  differ  as  a  function  of  the  participants’  level  of 
reported  video-game  experience.  Those  reporting  a  higher  level  of  video-game  experience 
performed  equally  well  under  both  test  conditions.  In  contrast,  those  reporting  a  lower  level  of 
video-game  experience  did  more  poorly  when  note-writing  and  directional  arrows  were  not 
available.  Their  performance  improved  significantly  when  these  organizational  tools  were 
available. 

Further  analysis  revealed  that  participants  who  reported  lower  video-game  experience  differed 
from  those  reporting  higher  video-game  experience  with  respect  to  memory  for  last-issued 
commands,  but  not  memory  for  aircraft  locations.  Memory  for  last-assigned  altitudes  was 
specifically  investigated  since  controllers  typically  record  altitude  changes  on  strips.  The  same 
pattern  of  results  was  obtained.  Participants  with  lower  video-game  experience  showed  poorer 
memory  for  altitude  changes  than  did  those  with  higher  experience,  particularly  when  note¬ 
writing  was  not  allowed.  Flight  strips  may  have  been  helpful  to  the  group  with  low  video-game 
experience  because  strip  marking  provided  a  means  by  which  aircraft  information  could  be  kept 
up-to-date  while  attention  was  devoted  to  the  task  of  manually  issuing  commands  and  receiving 
visual  feedback. 
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An  additional  important  finding  was  that,  regardless  of  their  level  of  video-game  experience, 
participants  believed  that  they  were  less  able  to  remember  the  call  signs  of  aircraft  in  this 
experiment,  in  which  control  commands  were  typed,  than  they  are  on  the  job,  in  which 
commands  are  issued  verbally.  Participants  recalled  an  average  of  82  percent  of  call  signs  in 
this  experiment.  One  participant  recalled  only  20  percent.  This  is  important  since  it  suggests 
that  critical  information  may  be  lost  under  conditions  in  which  keyboard  entries  are  used  to 
communicate  with  aircraft.  These  results  may,  therefore,  have  important  ramifications  for 
systems  requiring  keyboard  rather  than  verbal  communication. 

Several  important  implications  emerge  from  this  research:  1)  Controllers’  performance  with 
certain  ATC  systems  may  depend  upon  the  amount  of  prior  experience  they  have  had  with 
complex  systems,  like  video-games,  which  require  rapid  decision  making  and  skilled  eye-hand 
coordination,  2)  Memory  lapses  and  effective  memory-aiding  techniques  may  differ  between 
different  subgroups  of  controllers  depending  on  their  knowledge  of  and  skill  level  with  various 
systems,  3)  Results  of  research  conducted  with  low-fidelity  systems  can  not  be  directly 
translated  to  the  present  ATC  environment.  Although  controllers  were  willing  and  able  to  use 
a  low-fidelity  system,  testing  must  eventually  be  conducted  in  an  operational  environment 
because  no  simulation  completely  models  the  actual  system. 


1.  INTRODUCTION. 


1.1  PROLOGUE. 

Air  traffic  control  (ATC)  began  as  an  essential  response  to  increased  demands  on  the  airspace. 
During  the  early  years  of  ATC,  prior  to  and  including  World  War  II,  controllers  directed  traffic 
using  only  the  flight  plans  of  aircraft  and  a  clock.  Controllers  had  to  experiment  and  work  out 
methods  of  managing  their  responsibilities.  They  had  to  scan  flight  plans  to  gather  information, 
organize  the  information  in  order  to  make  decisions  and  plan  courses  of  action,  and  remember 
their  plans  and  actions  along  with  other  critical  information  to  maintain  separation  and  expedite 
the  flow  of  traffic. 

Technological  advancements  continue  to  be  made  to  meet  the  ever  increasing  demands  for 
service.  Numerous  hardware  and  software  tools  have  been  developed  to  improve  the  safety  and 
productivity  of  the  ATC  system.  Yet,  despite  all  the  automated  and  semi-automated  aids 
available,  controllers  must  still  make  use  of  the  same  skills  they  relied  on  previously.  They 
must  still  plan,  organize,  scan,  decide,  and  remember. 

The  ATC  system  remains  very  centered  on  the  human  controller.  While  controllers  add 
flexibility  and  adaptability  to  the  system,  they  also  add  the  potential  for  error,  as  Senders  and 
Moray  (1991)  have  described, 

"All  of  us  have  experienced  human  error.  When  we  interact  with  machines  or 
complex  systems,  we  frequently  do  things  that  are  contrary  to  our  intentions. 
Depending  on  the  complexity  of  the  system  and  the  intentions  of  the  people 
interacting  with  it,  this  can  be  anything  from  inconvenience  (often  it  is  not  even 
noticed)  to  a  genuine  catastrophe"  (Senders  and  Moray,  1991,  p.l). 

Identifying  the  human’s  role  in  the  system  and  the  potential  causes  of  human  error  is,  therefore, 
critical  in  order  to  enhance  system  safety  and  efficiency. 

An  administrator’s  task  force  (Operational  Error  Analysis  Work  Group,  1987)  studied  the 
frequency  and  possible  reasons  for  operational  errors  in  ATC.  An  operational  error  represents 
a  mistake  made  by  a  controller  which  fortunately  leads,  in  most  cases,  only  to  a  minor  violation 
of  airspace  separation  standards.  The  task  force  identified  visual  scanning  and  memory  as  two 
major  categories  of  errors.  The  latter  is  the  focus  of  this  report. 

Memory  requirements  in  ATC  are  demanding.  Air  traffic  controllers  are  surrounded  by  sources 
of  information  from  which  they  must  select  the  most  critical  components.  They  must  then  code 
and  store  this  data.  However,  this  is  not  always  done  effectively.  One  of  the  most  common 
expressions  uttered  by  controllers  who  have  made  an  operational  error  is:  "I  forgot!". 
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When  a  memory  lapse  occurs,  critical  information  about  aircraft  (e.g.,  location,  heading)  may 
be  temporarily  unavailable.  Without  ready  access  to  such  information,  situational  awareness 
may  not  be  adequately  maintained  and  a  crisis  may  result.  It  is  essential  to  understand  memory 
lapses  and  the  circumstances  under  which  they  occur  so  that  effective  training  procedures  and 
strategies  for  managing  memory  can  be  developed  to  assist  the  controller. 

In  1991,  there  were  701  controller  operational  errors  in  the  United  States  (FAA,  1992).  While 
this  is  promising  in  that  it  represents  a  reduction  from  872  errors  the  previous  year,  the  Federal 
Aviation  Administration  (FAA)  is  constantly  striving  to  eliminate  any  such  errors.  Efforts  to 
enhance  controller  memory  are  designed  to  assist  the  FAA  in  reaching  this  goal. 


1.2  BACKGROUND  OF  ATC  MEMORY  RESEARCH. 

Kinney,  Spahn,  and  Amato  (1977)  analyzed  FAA  reports  and  developed  8  categories  of  errors 
which  included:  controlling  in  another’s  airspace,  timing  and  completeness  of  flight  data 
handling,  inter-positional  coordination  of  data,  use  of  altitude  on  the  display,  procedures  for 
scanning  and  observing  flight  data,  phraseology  and  use  of  voice  communications,  use  of  human 
memory  to  include  relying  on  recall  in  a  noisy  environment,  and  dependence  on  automatic 
capabilities. 

The  FAA  uses  a  somewhat  different  set  of  categories  to  classify  operational  errors.  In  the 
profile  of  operational  errors  for  1987,  the  following  categories  were  employed:  radar  display, 
communication,  coordination,  aircraft  observation,  data  posting,  and  position  relief  (FAA, 
1988).  By  far  the  most  frequent  source  of  errors  identified  by  the  FAA  was  in  a  subclass  of 
"radar  display:  the  misuse  of  data."  It  is  likely  that  memory  issues  played  a  role  in  these  data- 
induced  errors. 

The  rate  at  which  information  flows  through  the  ATC  work  station  cannot  be  completely 
controlled  (Sperandio,  1971;  Kirchner  and  Laurig,  1971;  Thomas,  1985).  Controllers  must, 
therefore,  be  able  to  manage  memory  successfully  in  order  to  select  and  retain  all  of  the  critical 
elements  that  confront  them.  It  is  essential  to  establish  a  clear  understanding  of  controller 
memory  especially  as  new  hardware  and  software  systems  are  developed.  As  automation 
increases,  the  flow  of  information  is  likely  to  increase,  placing  an  even  heavier  burden  on 
controller  memory. 

The  amount  of  information  and  the  speed  with  which  it  can  be  processed  are  limited  (Finkleman 
and  Kirchner,  1980;  Spettel  and  Liebert,  1986;  Warm  and  Dember,  1986).  Opinions  concerning 
the  extent  of  human  limits  have  varied  considerably.  Miller’s  (1956)  concept  that  we  process 
about  7  (plus  or  minus  2)  chunks  of  information  at  any  one  time  has  become  accepted  doctrine, 
despite  the  fact  that  evidence  has  shown  otherwise  under  certain  conditions  (e.g.,  Klapp, 
Marshbum,  and  Lester,  1983).  The  "7  plus  or  minus  2"  view  may  be  too  restrictive  for  static 
memory,  on  which  it  was  based,  and  may  be  too  optimistic  for  dynamic  memory  (Moray,  1986). 
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It  is  likely  that  actual  working  memory  is  a  multi-operational  system  which  includes  static 
memory,  dynamic  memory,  and  attentional  components  (Baddeley,  1986). 

Working  memory  for  the  controller  certainly  includes  a  dynamic  component.  In  order  to 
manage  aircraft,  information  must  be  captured  and  retained  for  tactical  use  (3  to  5  minutes)  and, 
secondarily,  for  strategic  planning.  Each  aircraft’s  call  sign,  type,  route,  and  so  forth,  must  be 
retained  for  as  long  as  it  is  under  an  individual’s  control  and  then  discarded.  While  under 
control,  other  information  (e.g.,  altitude,  speed,  direction,  etc.)  must  be  continuously  updated 
and  readily  accessible  so  that  separation  of  aircraft  can  be  maintained.  Controllers’  memory 
requirements  are  further  burdened  by  inclement  weather  or  emergency  situations  since  these  may 
require  deviations  from  the  usual  expected  courses  of  action. 

In  today’s  ATC  system,  one  of  the  principal  tools  that  controllers  employ  to  keep  information 
up  to  date,  is  the  paper  flight  strip.  Controllers  are  required  to  annotate  these  strips  with 
changes  that  they  make  to  the  flight  plans  along  with  other  operational  considerations  (FAA, 
1989).  In  addition  to  writing  notes,  controllers  often  rearrange  strip  placement  to  act  as 
reminders  as  to  what  they  have  done  with  aircraft  and  what  they  will  need  to  do  in  the  future. 
The  important  connection  between  flight  strips  and  air  traffic  controllers’  (ATCSs’)  memory  has 
been  noted  by  Vortac  (1991).  This  report  indicated  that  "memory  is  essential  in  understanding 
the  relationship  between  flight  progress  strips  and  ATCS  performance.  However,  the 
relationship  may  not  be  immediately  obvious." 

The  value  of  flight  strips  has  recently  been  addressed  by  a  study  conducted  by  researchers  from 
the  Civil  Aeromedical  Institute,  the  University  of  Oklahoma,  and  the  FAA  Academy  (Vortac, 
Edwards,  Jones,  Manning  and  Rotter,  1992).  The  authors  noted  that,  although  controllers  often 
view  flight  progress  strips  as  unimportant,  they  do  use  them.  Their  study  focused  on  controller 
behavior  in  a  simulated  en  route  environment  and  found  that  note-writing  on  strips  was  one  of 
the  more  frequent  activities  that  controllers  engaged  in.  This  group  found  that  as  controllers 
became  busier  in  higher-complexity  scenarios,  they  fell  behind  in  updating  the  strips.  Further, 
controllers  increased  the  number  of  requests  for  information  from  pilots  in  higher-complexity 
scenarios,  implying  that  they  could  not  remember  or  retrieve  all  the  data  they  needed. 

One  important  aspect  of  flight  strip  management  is  that  it  allows  controllers  to  organize 
information,  enabling  it  to  be  recalled  more  efficiently.  The  important  relationship  between 
organization  and  memory  has  been  widely  reported  in  the  psychological  literature.  Bower, 
Clark,  Lesgold,  and  Winzenz  (1969),  for  example,  found  that  more  words  were  remembered  at 
time  of  test  if  they  were  initially  presented  according  to  an  organized  framework,  such  as  by 
category  (e.g.,  metals,  stones),  than  if  they  were  presented  randomly.  Benefits  are  also 
observed  when  the  organizational  scheme  is  self-imposed.  Those  who  organize  information 
more  extensively  have  been  found  to  recall  more  items  at  time  of  test  than  those  who  organize 
less  (Tulving,  1962). 

Means,  Mumaw,  Roth,  Schlager,  McWilliams,  Gagne,  Rosenthal,  and  Heon  (1988)  studied  the 
way  that  en  route  controllers  organized  aircraft.  They  observed  that  controllers  recalled  aircraft 
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in  groups,  invariably  drawing  one  group  at  a  time  when  tested.  When  asked  to  name  the 
groups,  controllers  labeled  them  in  accordance  with  a  specific  type  of  traffic  issue  (i.e.,  arrivals 
or  crossing  traffic  at  a  specific  fix).  Geographical  proximity  played  less  of  role  in  grouping  than 
did  the  interaction  and  potential  conflicts  between  members  of  a  group.  Organization  of 
information  has  been  identified  as  the  one  factor  which  has  the  greatest  probability  of  improving 
memory  performance  in  ATC  (Vortac,  1991). 

Activities  like  note-writing  and  other  flight  strip  management  techniques  (e.g.,  rearranging)  may 
be  important  to  memory  for  another  reasons  as  well.  There  is  a  finite  possibility  that  motoric 
enactment,  the  physical  manipulation  of  something  like  flight  strips,  may  be  the  key  to 
remembering  future  planned  actions.  Memory  for  anticipated  actions,  or  prospective  memory, 
is  a  critical  component  of  ATC,  however,  very  little  research  has  been  accomplished  which 
sheds  any  light  on  it. 

Benefits  of  physical  activity  on  memory  have  been  found  in  other  domains.  Koriat,  Ben-Zur, 
and  Nussbaum  (1990)  and  Zimmer  (1986)  each  found  that  performing  action  phrases  such  as, 
"tear  up  a  sheet  of  paper"  and  "blow  up  the  balloon,"  enhanced  recall  of  those  phrases. 
Memory  for  phrases  whose  actions  were  only  imagined  was  not  as  high.  Activity  may  involve 
a  deeper  level  of  processing,  making  information  more  memorable  and  accessible,  as  Norman 
(1992)  recently  indicated.  By  comparing  drawing  to  taking  a  picture  of  a  scene,  Norman 
described  that,  "the  act  of  drawing  requires  a  degree  of  concentration  and  study  that  intensifies 
the  experience  (p.  8)". 

As  r'  -ages  in  technology  occur,  the  human  operator’s  role  in  the  system  will  be  altered.  In 
more  automated  systems,  t  is  likely  that  paper  flight  strips  will  be  eliminated  and  replaced  by 
electronic  media.  It  has  been  duly  noted  that  the  impact  of  automating  the  tasks  currently 
undertaken  with  paper  flight  strips  must  be  determined,  since  their  value  has  been  so  widely 
emphasized  (Garland,  Stein,  Blanchard,  and  Wise,  1992).  Hopkin  (1991),  writing  about  future 
automated  systems,  commented  that  paper  strips  may  well  serve  beyond  their  original  intended 
purpose.  Hopkin  (1991b)  has  suggested  that  strip  management  activities  assist  in  the 
maintenance  of  situational  awareness,  help  controllers  remember  performed  and  to-be-performed 
actions,  and  also  help  controllers  plan  strategies  for  directing  traffic  (Hopkin,  1991).  The 
proposed  removal  of  paper  flight  strips  has  raised  the  concern  that  controllers  will  be  more  likely 
to  lose  situational  awareness  since  active  involvement  with  them  will  be  eliminated  (Hopkin, 
1991a;  Jackson,  1989). 


1.3  MEMORY  RESEARCH  PROJECT  AT  THE  FAA  TECHNICAL  CENTER. 

The  memory  research  program  began  in  1988.  The  goals  of  the  program  were  to  conduct 
research  to  more  fully  examine  the  factors  affecting  controller  memory  and  to  identify  and 
develop  memory  aids  that  would  assist  controllers  on  the  job.  One  of  the  initial  efforts  in  this 
program  was  conducted  by  Vingelis,  Schaeffer,  Stringer,  Gromelski,  and  Ahmed  (1990).  They 
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examined  theoretical  concepts  of  memory  and  identified  a  workable  cognitive  model  for 
controllers  that  encompassed  memory  issues.  Vingelis  et  al.  adapted  a  model  originally 
developed  by  Rasmussen  (Rasmussen,  1987;  Rasmussen  and  Lind,  1982),  which  involves  levels 
of  functioning  from  skill-based  to  knowledge-based  domains.  Each  of  these  domains  has  its  own 
unique  sources  of  memory-induced  error.  Further,  Vingelis  et  al.  defined  controller  short  term 
or  working  memory  in  terms  of  its  functional  requirements  (including  attention  and  rehearsal), 
its  contents,  organizational  structure,  operational  capacity,  and  limitations. 

Another  component  of  the  memory  project  involved  the  development  of  the  Controller  Memory 
Handbook  (Stein  and  Bailey,  1989).  This  document  was  created  based  on  the  memory  literature 
as  applied  to  person-machine  systems.  It  was  tailored  to  ATC  primarily  through  the  skill  of  the 
co-author,  Jim  Bailey,  who,  in  addition  to  being  an  artist,  was  also  an  air  traffic  controller. 
This  handbook  combined  text  and  cartoon  graphics  in  an  attempt  to  transfer  some  key  principles 
of  memory  to  the  controller  community. 

As  a  follow-up  to  the  Controller  Memory  Handbook,  copies  were  sent  to  a  selected  number  of 
facilities  along  with  an  evaluation  questionnaire.  The  questionnaire  had  two  parts.  In  the  first 
part,  respondents  were  asked  to  rate  the  handbook.  In  the  second,  respondents  were  requested 
to  describe  how  they  handled  memory  on  the  job  in  their  facilities.  Results  indicated  that 
personnel  liked  the  handbook  and  found  it  useful,  although  some  controllers  felt  that  it  was  too 
basic  to  meet  their  needs.  An  even  more  significant  finding  was  the  willingness  controllers 
expressed  for  siting  both  the  nature  of  their  problems  and  the  techniques  they  used  to  deal  with 
them  (Stein,  1991). 

Respondents  identified  the  following  causative  issues:  coordination,  attention,  distraction, 
fatigue,  change,  overload,  and  position  relief  briefings.  There  was  some  overlap  between  these 
results,  those  of  Kinney,  Spahn,  and  Amato  (1977),  and  those  reported  by  the  office  of  aviation 
safety  (FAA,  1988).  Controllers  offered  many  tools  they  said  worked  for  them  to  reduce  the 
probability  of  memory  lapses.  Most  of  these  centered  on  what  they  would  call  "good 
housekeeping."  This  implies  the  use  of  effective  organizational  skills  and  consistent  adherence 
to  procedures,  which  controllers  are  theoretically  taught  to  do  but  do  not  always  do  in  practice. 

In  order  to  pursue  these  issues  in  more  depth,  a  two  step  research  procedure  was  instituted  in 
1991  (Gromelski,  Davidson,  and  Stein,  1992).  The  first  step  of  this  procedure  was  a  mail 
survey  of  facility  managers.  This  was  followed  up  by  face-to-face  interviews  with  170 
controllers  at  facilities  across  the  continental  United  States.  The  advantage  of  interview  over 
survey  is  that  the  interviewer  can  probe  and  follow  up  on  issues  that  may  be  short-changed  in 
a  survey  approach.  One  of  the  most  significant  findings  of  this  study  was  that  controllers  were 
aware  of  the  memory  issues  and  of  the  aids  available  within  the  system  as  it  exists  today. 
However,  they  resist  using  them  for  the  same  sorts  of  reasons  that  lead  them  to  be  reluctant  to 
ask  for  help  if  they  are  overloaded.  This  somehow  violates  the  controller  culture.  Controllers 
did  indicate  that  flight  strip  management  activities,  including  flight  strip  organization  (cocking, 
tilting)  and  marking  (keeping  notes),  were  the  mos.  used  memory-aiding  techniques.  They  also 
agreed  that  good  controllers  engage  in  certain  desirable  behaviors.  Good  controllers  were 
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reported  to  preplan  actions,  prioritize  work  sequences,  organize  aircraft  information,  anticipate 
future  states  or  problems,  and  use  effective  communication. 

Zingale,  Gromelski,  and  Stein  (1992)  examined  some  of  these  concerns  in  a  laboratory  setting 
in  which  systematic  control  was  possible.  These  were  the  first  experiments  in  a  series  leading 
to  the  research  described  in  this  report.  Zingale  et  al.  worked  with  college  students  who  were 
studying  to  be  pilots  and  had  no  background  in  ATC.  They  were  taught  the  principles  of  control 
and  were  tested  using  a  personal  computer  (pc)-based  simulation,  TRACON  II.  TRACON  II 
was  originally  designed  as  a  computer  game,  and  part  of  the  purpose  of  these  studies  was  to 
evaluate  the  feasibility  of  using  it  as  a  tool  to  study  control  performance  and  memory  issues. 

In  one  experiment,  participants  were  either  encouraged  or  discouraged  from  developing  operating 
strategies  in  advance  of  controlling  traffic.  Results  indicated  that  this  had  little  impact  on 
performance.  In  a  second  experiment,  the  availability  of  planning  time  prior  to  working 
simulated  traffic  (2  minutes  vs.  5  minutes)  was  tested.  This  also  did  not  make  a  difference. 

In  a  third  experiment,  participants  were  tested  for  recall  of  critical  flight  information  after  being 
given  the  opportunity  to  mark  flight  progress  strips.  Participants  were  instructed  to  record  the 
commands  they  issued  to  each  aircraft  on  flight  strips  as  each  command  was  executed. 
Performance  was  also  monitored  as  participants  proceeded  through  the  test  session  and  an  overall 
performance  score  was  obtained.  To  test  memory  for  critical  information,  the  flight  strips  were 
removed  at  the  end  of  the  session  and  participants  were  instructed  to  report  all  of  the  commands 
they  had  issued  to  each  aircraft.  Some  participants  used  flight  strips  more  extensively  than 
others,  recording  all  or  nearly  all  of  the  commands  issued,  while  others  recorded  only  a  portion 
of  them.  It  was  found  that  those  who  wrote  more  on  the  strips  remembered  more  commands 
and  also  performed  better.  In  addition,  those  who  wrote  more  on  the  strips  reported  lower 
workload  levels  than  those  who  wrote  less.  It  was  recognized  that  the  decision  to  write  or  not 
may  well  have  been  associated  with  what  each  participant  brought  with  him/her  to  the  study, 
including  basic  abilities  and  self  confidence.  One  of  the  most  noteworthy  findings  in  this  series 
of  experiments,  was  that  the  potential  for  using  low-fidelity  PC-based  ATC  simulations  in  testing 
was  demonstrated. 

Since  the  results  of  the  third  experiment  suggested  that  flight  strip  management  may  be  important 
to  memory  and  performance  in  ATC,  an  additional  experiment  was  conducted  to  evaluate  the 
effects  of  note-writing  and  flight  strip  organization  on  situational  awareness  and  control  ability 
with  actual  air  traffic  controllers.  (Further  pilot  work  was  also  conducted  with  non-controllers 
prior  to  this.  The  results  of  this  work  are  described  in  appendix  A.  The  apparent  differences 
between  the  results  obtained  for  controllers  and  non-controllers  suggests  that  research  on  ATC 
issues  with  non-controllers  must  be  interpreted  with  caution.  Data  from  non-controllers  alone 
may  not  adequately  reflect  the  performance  characteristics  of  controllers,  and  may  lead  to 
incomplete  or  inaccurate  conclusions.) 
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2.  EXPERIMENT:  USEFULNESS  OF  FLIGHT  STRIPS  (NOTE-WRITING  AND 
ORGANIZATIONS  TO  SITUATIONAL  AWARENESS  AND  PERFORMANCE. 


2.1  PURPOSE. 

The  purpose  of  the  current  experiment  was  to  determine  the  usefulness  of  note-writing  and  flight 
strip  organization  to  controller  situational  awareness  and  control  ability.  Maintaining  situational 
awareness  involves  the  ability  to  access  critical  information  about  aircraft,  such  as  aircraft 
location  and  current  status  (e.g.,  altitude,  speed).  A  loss  of  situational  awareness,  forgetting 
about  an  aircraft  or  what  actions  need  to  be  taken,  can  result  in  severe  consequences  such  as 
separation  conflicts  or  crashes. 

The  results  of  a  prior  experiment,  described  above,  indicated  that  note-writing  on  flight  strips 
was  associated  with  improved  memory  for  critical  information  (i.e.,  commands  issued)  and 
improved  ATC  performance  (Zingale,  Gromelski,  and  Stein,  1992).  The  current  experiment 
further  investigated  the  effect  of  flight  strip  manipulation  by  investigating  note-writing  and  strip 
organization  on  control  performance  and  situational  awareness.  In  addition  to  keeping  notes, 
participants  in  this  experiment  were  provided  with  arrows  on  flight  strips  indicating  that  aircraft 
were  either  arrivals  ( * ),  departures  ( t ),  or  overflights  («*).  These  were  included  to  provide  an 
additional  tool  to  assist  participants  in  their  organization  of  the  aircraft.  Situational  awareness 
was  assessed  by  testing  each  participant’s  knowledge  of  aircraft  position  and  the  last  command 
issued  to  each. 


2.2  METHOD. 


2.2.1  Participants. 

Participants  consisted  of  air  traffic  controllers  from  the  FAA  Technical  Center  and  the  Atlantic 
City  International  Airport  Tower.  They  were  assured  of  complete  anonymity.  A  total  of  8 
controllers,  6  from  the  tower  and  2  from  the  Technical  Center,  participated.  All  of  the 
participants  had  at  least  4  years  of  ATC  experience  in  a  terminal  environment  (mean =10.38, 
standard  deviation  (SD)  =  5.63).  Three  of  them  reported  having  prior  experience  with  the 
TRACON.  All  of  the  controllers  had  worked  with  flight  progress  strips  on  the  job,  and  used 
strip  board  management  techniques  routinely,  although  some  expressed  that  they  used  the  strips 
reluctantly.  Participants  received  overtime  pay  for  their  participation. 

Participants  rated  themselves  (1  =  lowest,  10= highest)  in  several  areas  that  were  thought  could 
potentially  relate  to  performance.  Controllers  rated  themselves  low  in  terms  of  computer 
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experience  (mean =3.38,  SD=2.39)  and  video-game  experience  (mean =3.75,  SD=2.12). 
Levels  of  reported  vision  (mean =9. 63,  SD  =  .74)  and  health  (mean =10.00,  SD=0)  were  high. 
Controllers  also  indicated  that  they  freely  volunteered  to  participate  (mean =10.0,  SD=0),  but 
varied  in  their  motivational  level  (mean =3. 38,  SD=3.54).  (The  question  on  motivation  was 
worded  so  that  a  response  of  1  was  highest  and  10  lowest).  Stress  levels  for  this  group  were 
generally  low  to  moderate  (mean =4.25,  SD  =  1.67). 


2.2.2  Equipment. 

The  TRACON  II  ATC  Simulator  for  the  IBM  PC  (Wesson  and  Young,  1990)  was  used  in 
testing.  Simulations  were  run  on  486  computers  at  the  Human  Factors  Laboratory  of  the  FAA 
Technical  Center.  TRACON  II  is  capable  of  presenting  aircraft  in  sectors  surrounding  major 
terminal  control  areas  like  Boston,  Los  Angeles,  etc.  The  areas  represented  are  limited  only  by 
the  need  for  site-specific  data.  Sector,  number  of  aircraft,  weather  conditions,  pilot 
performance,  equipment,  and  number  of  potential  emergency  situations  can  be  specified  by  the 
user.  Scenarios  can  be  generated  randomly  by  the  game  itself  or  specifically  programmed. 
Programmed  scenarios  were  used  in  this  study  to  control  for  the  variability  that  different 
scenarios  may  produce. 

Programmed  scenarios  were  created  using  the  DOS  editor  as  described  by  the  TRACON  II 
manual.  In  each  of  these  scenarios,  the  number  of  relevant  airports  and  fixes  within  the  sector 
was  reduced  so  that  participants  would  be  able  to  learn  the  names  and  locations  quickly,  enabling 
training  to  proceed  more  rapidly.  The  Los  Angeles  sector,  including  two  of  its  five  airports 
(LAX  and  Long  Beach),  and  7  of  15  outer  fix  locations,  was  selected.  Weather  and  equipment 
were  set  to  "perfect".  Pilot  performance  was  set  to  "average"  for  smaller  aircraft,  so  that 
participants  would  have  to  pay  attention  to  readbacks  to  ensure  that  commands  had  been 
accepted.  Aircraft  were  included  to  cover  a  range  of  types.  The  times  at  which  the  aircraft 
entered  the  sector  were  distributed  so  that  eight  to  nine  aircraft  would  be  present  on  the  scope 
at  8  minutes  and  at  16  minutes  into  the  scenario.  These  were  the  times  at  which  the  scenarios 
would  be  stopped  and  participants  would  be  tested  on  their  situational  awareness.  It  is  important 
to  note  that  the  number  of  aircraft  present  in  the  scenario  at  any  given  time  cannot  be  completely 
controlled  by  the  experimenters  since  the  ability  of  participants  to  successfully  manage  the  flow 
of  traffic  into  and  out  of  the  airspace  affects  this  variable.  This  is  a  characteristic  of  a  free-play 
simulation. 

Training  scenarios  were  developed  and  varied  in  complexity.  Early  in  familiarization  training, 
fewer  aircraft  and  fewer  conflict  situations  were  included.  As  training  progressed,  scenarios 
increased  in  complexity  to  contain  the  number  of  aircraft  that  would  be  present  in  the  test 
scenarios.  Each  test  condition  used  the  same  scenario.  However,  the  aircraft  call  signs  differed 
between  them.  This  allowed  for  a  standardized  test  environment  but  made  it  so  that  participants 
would  be  less  able  to  recognize  the  scenarios  as  identical.  Test  scenarios  included  14  aircraft, 
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8  arrivals,  4  departures,  and  2  overflights.  The  majority  of  aircraft  were  arrivals  since  these 
required  more  commands  to  manage  successfully. 

The  version  of  TRACON  used  in  this  experiment  allows  the  user  to  specify  whether  to  exit  the 
program  following  a  "crash",  defined  as  a  separation  of  aircraft  of  less  than  1/2  mile 
horizontally  or  500  feet  vertically.  This  option  was  turned  off  so  that  the  scenario  would  resume 
after  a  crash  message,  allowing  data  to  be  collected  while  the  session  was  carried  to  completion. 
No  crashes  occurred  during  the  test  sessions. 


2.2.3  Training. 

Participants  were  first  provided  with  a  brief  synopsis  of  the  project’s  background  and  the  overall 
intent  of  the  work.  Prior  to  receiving  hands-on  training,  participants  were  provided  with  an 
overview  to  illustrate  the  basics  of  TRACON.  This  included  a  demonstration  by  one  of  the 
researchers,  a  former  ATCS.  Participants  were  able  to  observe  the  way  in  which  aircraft  were 
handled,  including  how  key  functions  are  used  to  issue  commands  and  the  way  in  which  errors 
(crashes,  separation  conflicts,  handoff  errors,  missed  approaches)  are  reported.  The  scenarios 
were  frequently  paused  so  that  participants  would  have  the  opportunity  to  ask  questions.  It  was 
realized  that  the  method  by  which  commands  are  issued  in  TRACON  II  is  different  from  the  way 
in  which  controllers  issue  them  on  the  job.  Controllers  normally  communicate  with  aircraft 
verbally.  This  difference  was  obviously  noted,  and  some  of  the  controllers  indicated  that  it 
would  interfere  with  their  operating  abilities.  This  aspect  of  controllers’  performance  is 
addressed  further  in  the  Results  and  Discussion  sections. 

Participants  were  provided  with  a  detailed  training  manual  which  included  a  paper  map  of  the 
airspace  indicating  the  names  and  locations  of  all  fixes  and  airports  (see  appendix  B,  page  B-8). 
In  addition,  a  "quick  reference  card"  was  provided  which  described  all  command  entries  (see 
appendix  B,  page  B-12).  Six  keys  (insert,  home,  page  up,  delete,  end,  and  page  down)  and  the 
up,  down,  left  and  right  arrow  keys  are  used  to  issue  most  TRACON  commands.  Commands 
were  typed  onto  colored  labels  and  placed  over  the  appropriate  keys  to  facilitate  learning  and 
training  (see  appendix  B,  pages  B-10  and  B-ll  for  illustration). 

Instruction  was  given  for  each  type  of  aircraft,  arrivals,  departures,  and  overflights.  The 
training  emphasized  efficient  ways  of  managing  the  different  types  of  aircraft  so  that  minimal 
keyboard  entries  would  be  required.  This  was  done  to  allow  less  skilled  typists  to  focus  on  the 
job  of  controlling  traffic  rather  than  on  keystrokes.  Departures  and  overflights  required  minimal 
control  instructions  unless  a  conflict  situation  was  pending.  Arrivals  were  the  most  demanding. 
To  reduce  the  number  of  keystrokes  used  to  control  arrivals,  the  necessary  control  operations 
were  described  as  a  stepwise  procedure.  Following  initial  radar  contact,  one  step  involved 
sending  the  aircraft  to  a  fix  location  just  outside  the  approach  area  of  the  airport,  a  second 
involved  descending  the  aircraft  to  its  approach  altitude,  and  the  third  involved  clearing  the 
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aircraft  for  final  approach.  This  technique  involved  less  use  of  vectoring,  and  thereby  reduced 
the  number  of  necessary  keyboard  entries. 

Participants  were  also  instructed  as  to  how  to  issue  multiple  commands  to  aircraft  and  on  the  use 
of  data  tags.  These  tags  included  information  (usually  a  destination)  which  could  be  typed  in 
and  placed  under  an  aircraft’s  call  sign  on  the  scope.  Participants  were  given  the  option  to  use 
them  as  needed.  However,  each  was  encouraged  to  follow  his/her  own  plan  of  action  if  it  would 
better  assist  control  of  the  traffic.  Participants  were  told  that  their  strategies  would  be  a  focus 
of  the  research  and  that  they  would  be  asked  to  indicate  the  way  in  which  they  handled  aircraft 
after  each  test  session. 

Each  of  the  hands-on  training  scenarios  was  programmed  to  run  for  approximately  1/2  hour. 
Duration  of  the  scenarios  varied  from  individual  to  individual  since  their  completion  depended 
on  the  timely  and  accurate  manner  in  which  aircraft  were  handled.  During  earlier  training 
sessions,  these  scenarios  typically  ran  longer  since  participants  were  less  efficient  at  managing 
the  traffic.  The  training  and  test  scenarios  shared  many  of  the  same  types  of  potential  conflict 
situations,  so  increased  exposure  enabled  participants  to  be  better  prepared  to  anticipate  problems 
and  to  practice  taking  preventive  measures. 

Participants  were  monitored  throughout  training  to  ensure  that  they  were  working  effectively  and 
to  allow  them  an  opportunity  to  ask  questions.  There  were  two  basic  stages  of  training.  The 
first  stage  involved  hands-on  experience  with  TRACON  using  all  of  the  available  computer- 
generated  information,  including  computer-generated  flight  strips.  The  second  stage  involved 
working  through  scenarios  while  using  paper  flight  strips  similar  to  those  used  in  testing. 
During  these  training  scenarios,  the  computer-generated  active  flight  strips  were  blocked  from 
view.  Participants  practiced  writing  on  flight  strips  and  also  practiced  using  flight  strips  that 
they  could  only  look  at  and  not  write  on.  Participants  were  provided  with  additional  practice, 
as  needed. 

The  amount  of  training  time  necessary  to  become  proficient  with  the  control  commands,  airspace 
configuration,  basic  control  strategies,  and  separation  standards  varied  with  the  individual.  In 
general,  participants  who  were  computer  literate  and/or  were  proficient  typists,  tended  to  learn 
at  a  faster  rate  than  those  who  were  less  familiar  with  keyboard  layout. 

Near  the  end  of  training,  participants  were  informed  that  the  test  sessions  would  involve  a  test 
of  their  memory  for  aircraft  information  when  the  screen  was  blocked  from  view.  They  were 
asked  to  consider  how  they  would  be  able  to  manage  the  traffic  if  the  "radar  went  out".  They 
were  then  given  a  brief  sample  of  what  this  test  environment  would  be  like. 

Controllers  had  restrictive  schedules.  They  were  available  for  a  total  of  8  hours.  Six  of  these 
participants  had  to  complete  training  and  testing  over  the  course  of  1  day.  Training  took  place 
during  a  3-hour  morning  time  block  and  testing  took  place  in  the  afternoon.  The  other  two 
participants  were  available  for  each  of  two  mornings  which  enabled  training  to  take  place  1  day 
and  testing  the  next. 
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2.2.4  Test  Conditions. 


Participants  were  each  tested  under  two  experimental  conditions.  In  one  condition,  they  were 
required  to  write  notes  on  the  strips  while  they  were  working.  These  strips  additionally 
contained  arrows  indicating  whether  aircraft  were  arrivals  ( l ),  departures  ( t ),  or  overflights  (**) 
as  a  further  organizational  tool.  In  the  second  condition,  participants  were  not  allowed  to  write 
notes  on  the  strips.  Organization  of  the  strips  was  also  reduced  by  deleting  the  arrows. 

Participants  were  provided  with  flight  strips  in  separate  strip  holders  (see  appendix  C,  page  C-l 
through  C-2  for  sample  flight  strips).  This  procedure  was  used  to  provide  controllers  with  a 
familiar  work  setting  to  minimize  any  effects  of  negative  transfer  to  the  test  environment. 
Allowing  participants  to  work  with  separate  flight  strips  enabled  analyses  to  be  conducted  with 
respect  to  different  flight  strip  management  techniques  used  in  the  test  conditions. 


2.2,5  Performance  Measures. 

Participant’s  were  evaluated  on  situational  awareness  (SA)  and  control  performance  (SCORE) 
in  each  condition.  SA  was  tested  during  2  intervals  in  which  the  on-going  scenario  was  blocked 
from  view.  The  scenario  was  "paused"  and  the  display  completely  covered  by  a  sheet  of  opaque 
white  paper.  This  occurred  at  approximately  8-9  and  16-18  minutes  into  the  scenario. 
Participants  were  not  informed  as  to  when  the  SA  tests  would  occur.  At  these  two  test  points, 
participants  were  instructed  to  report  the  locations  of  each  of  the  aircraft  currently  under  their 
control  and  to  report  the  last  command  issued  to  each.  A  map  of  the  airspace  and  a  list  of  the 
aircraft  call  signs  were  provided  for  them  to  indicate  their  responses  (see  appendix  C,  pages  C-3 
through  C-5).  The  map  provided  minimal  reference  information  and  did  not  include  fix  names. 

SA  was  evaluated  by  determining  the  proportion  of  correct  responses  regarding  aircraft  location 
and  last-issued  commands.  Actual  aircraft  locations  were  determined  by  uncovering  the  display 
screen  while  still  in  pause  mode  and  recording  each  aircraft’s  placement  on  the  map  containing 
participant  responses.  Aircraft  location  reports  were  scored  in  the  following  way:  one  full  point 
was  given  if  an  aircraft  was  identified  and  placed  correctly  on  the  map.  Correct  placement 
meant  that  it  was  within  5  miles  (1  inch)  of  its  actual  location.  One-half  point  was  given  if  the 
location  indicated  was  within  5  to  7  miles  of  the  actual  location,  and  zero  points  were  given  if 
the  location  indicated  was  beyond  this  limit.  One-quarter  point  was  given  if  a  participant 
indicated  that  an  aircraft  was  present  in  a  particular  location  but  could  not  report  which  aircraft 
it  was. 

To  indicate  aircraft  location,  participants  were  required  to  mark  the  map  with  an  aircraft 
identifier,  ideally  the  aircraft  call  sign.  However,  in  TRACON  the  call  sign  does  not  have  the 
same  significance  as  it  does  in  actual  ATC  because  it  is  not  spoken  each  time  a  command  is 
issued.  Instead,  participants  use  the  keyboard’s  arrow  key  to  move  a  cursor  to  highlight  an 
aircraft.  Once  highlighted,  the  commands  can  be  issued  to  that  aircraft  via  the  keyboard.  Since 
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the  call  signs  were  not  as  integral  a  part  of  the  command  sequence  in  TRACON,  partial  credit 
(one-half  point)  was  given  if  the  participants  were  able  to  mark  the  appropriate  location  with  a 
distinguishing  identifier  (i.e.,  the  aircraft  from  Anaheim  (AHEIM)  going  to  Long  Beach  at  4000 
feet). 

Memory  for  last-issued  commands  for  each  identified  aircraft  was  assessed  by  examining  the 
written  reports  of  participants.  Participants  were  able  to  refer  to  the  sheet  containing  the  aircraft 
call  signs  and  recorded  the  last  command  issued  next  to  each.  In  cases  in  which  the  call  sign 
could  not  be  paired  with  an  aircraft,  participants  were  told  to  use  other  means  to  identify  the 
aircraft.  For  example,  they  could  indicate  the  location  of  an  aircraft  (i.e,,  the  one  from  SAUGS 
to  Long  Beach)  and  that  the  last  command  issued  to  that  aircraft  was  to  descend  to  1600  feet. 

Performance  was  assessed  by  using  a  modified  version  of  each  participant’s  TRACON-provided 
score  (appendix  C,  page  C-6  for  actual  TRACON  scoring  system).  TRACON  scores  are 
tabulated  by  adding  points  for  each  successfully-completed,  significant  action,  such  as  landing 
an  aircraft  and  handing  off  an  aircraft  to  the  next  sector.  Points  are  deducted  for  missed 
approaches,  missed  handoffs,  separation  conflicts,  and  crashes.  Additionally,  a  smaller  number 
of  points  is  subtracted  whenever  any  command  is  issued  and  if  substantial  delays  occur  between 
control  actions.  Maximum  point  scores  are  based  on  the  number  of  aircraft  in  the  scenario  and 
on  the  size  and  type  of  the  aircraft  included.  Larger  aircraft  have  more  point  values  associated 
with  them  than  smaller  aircraft.  Therefore,  different  scenarios  have  different  maximum  possible 
scores.  A  maximum  of  12780  points  was  possible  in  the  scenarios  used  in  this  experiment. 

The  TRACON  scores  alone  can  be  misleading  as  a  reflection  of  true  performance,  however, 
since  it  is  possible  for  a  participant  to  obtain  a  relatively  high  score  while  taking  a  very  long 
time  to  complete  the  scenario.  This  would  indicate  that  the  aircraft  were  not  handled  efficiently. 
Since  this  aspect  of  performance  is  not  adequately  reflected  in  the  overall  score,  a  new 
performance  score  was  derived.  This  score  was  obtained  by  dividing  the  TRACON  score  by 
the  time  taken  to  complete  the  scenario.  This  "points  per  minute"  score  more  accurately 
reflected  a  participant’s  efficiency  in  working  traffic  and  is  the  performance  "SCORE"  referred 
to  throughout  the  remainder  of  the  report. 


2.2.6  Procedure. 

Prior  to  the  start  of  the  training  and  test  sessions,  participants  filled  out  a  questionnaire  designed 
to  identify  factors  that  may  have  additionally  influenced  performance,  such  as  amount  of  prior 
experience  with  TRACON  or  extent  of  ATC  experience  (appendix  D,  pages  D-l  through  D-3. 

Participants  were  tested  individually  so  that  a  record  of  all  of  their  actions  could  be  accurately 
maintained  by  the  experimenters.  Recordings  of  every  command  were  kept  by  category  as 
shown  by  the  example  in  appendix  D,  pages  D-4  through  D-5.  The  sequence  of  commands  for 
each  aircraft  was  also  noted  on  these  sheets.  The  sessions  were  also  video-taped  for  seven  of 
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the  controllers.  Video-tape  recordings  were  made  of  the  computer  screen,  showing  aircraft 
positions,  aircraft  movements,  commands  issued,  and  feedback  messages,  and  of  the  participant’s 
flight  strip  management  activities.  No  full-face  video  tapes  were  made  in  order  to  protect  the 
identities  of  the  participants.  Participants  were  fully  informed,  prior  to  testing,  that  these  events 
would  be  recorded. 

Experimental  sessions  were  1-1/4  to  1-1/2  hours  long  including  the  intervals  during  which  SA 
was  assessed  and  the  time  taken  to  complete  a  post-session  questionnaire.  Both  test  sessions 
were  conducted  on  the  same  day.  A  break  of  10  to  15  minutes  separated  these  sessions  to 
reduce  fatigue  as  much  as  possible  given  these  circumstances. 

Prior  to  the  start  of  the  first  experimental  session,  participants  were  provided  with  about  5 
minutes  of  "warm  up"  time  on  a  short  scenario.  Participants  worked  in  this  short  scenario  in 
the  same  manner  that  they  would  in  the  subsequent  test  condition,  either  writing  or  not  writing 
on  the  paper  strips.  The  test  session  started  immediately  afterwards.  Participants  were  informed 
that  the  scenario  would  be  blocked  from  view  at  various  times  throughout  the  session  but  were 
not  told  when  this  would  occur.  They  were  also  told  that  they  would  be  asked  to  indicate  the 
location  of  each  aircraft  and  the  last  command  issued  to  each,  and  that  during  the  test  interval 
the  paper  flight  strips  would  not  be  available  to  them. 

Participants  were  given  some  time  to  review  and  rearrange  the  flight  strips  before  beginning  the 
scenario.  Flight  strips  were  placed  to  the  right  of  the  keyboard  because  all  of  the  controllers 
tested  were  right-handed.  One  of  the  two  researchers  (both  of  whom  were  also  authors  of  the 
report)  sat  to  the  left  and  slightly  behind  each  participant  to  be  able  to  observe  each  of  the 
commands  being  issued.  The  researchers  maintained  minimal  interaction  with  the  participants 
as  they  were  working.  Researchers  recorded  each  command  as  it  was  issued  on  the  scoring 
sheets  described  above. 

At  the  first  stop  point,  (approximately  8  minutes  into  each  scenario),  the  display  was  "paused" 
by  the  experimenter  and  was  covered  from  participants’  view.  Participants  were  asked  to  briefly 
review  their  list  of  active  aircraft  on  their  flight  strips  while  the  researchers  noted  on  the  scoring 
sheets  the  last  command  issued  for  each  aircraft.  The  participant’s  flight  strips  were  then 
removed  and  they  were  given  the  map  and  the  list  of  aircraft  names.  Participants  had  as  much 
time  as  they  felt  necessary  to  indicate  all  that  they  could  about  aircraft  locations  and  commands. 
They  could  complete  the  information  in  any  order  they  chose.  When  completed,  the  display  was 
uncovered,  and  while  still  in  pause  mode,  the  actual  locations  of  the  aircraft  were  recorded  on 
the  airspace  map.  The  list  of  call  signs  containing  the  last-issued  commands  were  collected  for 
subsequent  scoring.  Participants  were  then  allowed  whatever  time  was  necessary  to  bring 
themselves  up  to  date  before  resuming  the  scenario.  At  the  second  stop  point,  (approximately 
16  minutes  into  the  scenario),  the  display  was  paused  again  and  this  procedure  was  repeated. 
Following  this  second  test  interval,  the  scenario  was  run  to  completion. 

The  scoring  information  provided  by  TRACON  was  recorded  immediately  after  the  scenario 
ended  (see  appendix  D,  page  D-6).  Participants  also  indicated  their  perceived  level  of  workload 
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for  the  session  using  the  12-point  scale  at  the  bottom  of  this  sheet,  with  1  indicating  the  lowest 
and  12  the  highest  possible  workload  level.  Participants  also  completed  a  post-session 
questionnaire  following  each  of  the  experimental  sessions  in  order  to  provide  additional 
information  as  to  what  may  have  affected  their  performance  (see  appendix  D,  pages  D-7  through 
D-9).  In  addition,  the  experimenters  discussed  performance  with  participants  to  elicit  more 
information  about  their  strategies. 


2.2.7  Design. 

A  2  x  2  mixed  design  was  using  in  which  TEST  CONDITION  (WRITING  vs.  NO  WRITING) 
served  as  the  within-subjects  factor  and  TEST  ORDER  (WRITING  condition  first  or  second) 
served  as  the  between-subjects  factor  (see  table  2-1).  The  order  for  testing  participants  was 
counterbalanced  to  reduce  the  confounding  effect  of  practice  on  the  conditions  under 
investigation.  SA  and  performance  (SCORE)  served  as  dependent  variables.  In  all  analyses, 
the  two  dependent  variables  were  analyzed  separately  in  keeping  with  the  distinction  proposed 
by  Fndsley  (1989).  She  argued  that  situational  awareness  should  be  considered  separately  from 
decision  making  and  performance  because  it  is  possible,  for  example,  to  achieve  high  situational 
awareness  but  not  have  other  skills  necessary  to  properly  formulate  decision  strategies  or  carry 
out  appropriate  actions. 


TABLE  2- 1 .  EXPERIMENTAL  DESIGN 


WRITING 

NO  WRITING 

ORDER  =  1 

2nd 

1st 

ORDER  =  2 

1st 

2nd 

Note.  Participants  worked  under  both  the  WRITING  and  the  NO  WRITING 
conditions  in  the  order  indicated. 


These  data  were  analyzed  using  a  statistical  package  for  the  pc,  the  Statistical  Package  for  Social 
Sciences  (SPSS).  Analysis  of  variance  (ANOVA),  a  statistical  procedure  used  to  determine 
whether  the  differences  between  two  or  more  means  are  significant,  was  used  to  evaluate 
differences  in  performance  scores  and  SA  scores  as  a  function  of  CONDITION,  ORDER,  and 
the  CONDITION  x  ORDER  interaction.  A  multiple  regression  analysis  was  conducted  on  the 
preliminary  questionnaire  variables  and  performance  scores.  Similarly,  a  multiple  regression 
analysis  was  conducted  on  the  preliminary  questionnaire  variables  and  SA  scores.  Partial 
correlations  were  obtained  from  these  analyses  to  determine  the  relationship  between  each 
independent  variable  (i.e.,  level  of  stress)  and  each  dependent  variable  (i.e.,  performance).  This 


14 


procedure  partitions  out  the  combined  effects  of  correlated  independent  variables  (i.e.,  stress  and 
fatigue)  on  the  dependent  variable,  so  that  only  the  relationship  of  individual  variables  is 
determined. 

Other  ANOVA’s  were  conducted  to  evaluate  differences  between  test  conditions  on  post-session 
questionnaire  variables,  (e.g.,  level  of  perceived  workload,  level  of  stress).  The  relationship 
between  various  strategies  and  performance  was  also  investigated.  The  proportions  of  several 
command-types  (e.g.,  altitude  changes,  speed  changes)  were  analyzed  as  a  function  of  test 
condition  to  determine  whether  certain  commands  were  used  more  in  one  condition  or  another. 

An  analysis  of  the  video-tapes  was  also  conducted  to  determine  whether  flight  strip  management 
differed  between  test  conditions.  The  number  of  non-writing  actions  (arranging  strip  placement 
or  cocking  strips  left/ right)  was  determined  for  each  test  condition  and  compared. 


3.  RESULTS. 

Controllers’  SA  did  not  differ  significantly  between  test  conditions,  F(l,6)  =  1.05,  P>  .05  (see 
figure  3-1).  In  addition,  neither  the  effect  of  ORDER,  F(  1 ,6)  = .  1 8,  p>  .05,  nor  the  interaction 
of  ORDER  x  CONDITION,  F(l,6)  =  .12,  p>  .05,  were  significant.  SCORE  also  did  not  differ 
significantly  as  a  function  of  CONDITION,  ORDER,  or  their  interaction,  p>  .05,.  An  analysis 
of  covariance  (ANCOVA)  was  also  conducted  on  these  data  with  the  ORDER  variable  as  the 
covariate.  No  significant  differences  were  found  (p>.05). 


Note.  Standard  deviations  above  each  data  point. 


FIGURE  3-1.  MEAN  SA  SCORE  AS  A  FUNCTION  OF  CONDITION  AND  TEST  ORDER. 
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In  addition  to  the  performance  measures  described,  post-session  questionnaire  responses  were 
examined  to  determine  whether  participants  reported  any  differences  between  operating  under 
the  WRITING  condition  and  the  NO  WRITING  condition.  Each  participant  provided  an 
assessment  of  his/her  performance  (ASSESS),  and  gave  ratings  of  perceived  levels  of  workload 
(WORKLOAD),  busyness  (BUSY),  thinking  and  concentration  (THINK),  stress  (STRESS),  and 
fatigue  (FATIGUE).  Participants  also  reported  how  helpful  they  felt  the  flight  strips  were  in 
each  condition.  They  rated  how  much  the  strips  assisted  their  control  performance  (CONTROL) 
and  how  much  the  strips  assisted  their  SA  (MEMORY).  Ratings  were  provided  using  a  ten-point 
scale  in  which  one  was  the  lowest  and  ten  was  the  highest  response  level.  Workload,  however, 
was  reported  using  a  12-point  scale. 

Analyses  of  the  post-session  questionnaire  variables  indicated  that  the  controller’s  self-assessment 
ratings  were  found  to  differ  significantly  between  test  conditions,  F(l,6)  =  10.71,  p<  .05.  They 
reported  that  their  performance  was  poorer  under  the  WRITING  condition.  The  effect  of 
ORDER  was  not  significant,  F(l,6)  =  .08,  p>.05,  however,  the  ORDER  x  CONDITION 
interaction  was  significant,  F(1 ,6)=  10.71 ,  p<.05.  Planned  contrasts  indicated  that  controllers 
who  worked  under  the  NO  WRITING  condition  first,  reported  a  greater  perceived  difference  in 
their  performance  between  the  two  conditions,  F(l,6)=6.97,  p<.05. 

For  the  FATIGUE  variable,  neither  main  effect  was  significant  (p>  .05),  however,  the  ORDER 
x  CONDITION  interaction  was  significant,  F(I,6)=1 1.71,  p<.05.  Contrasts  indicated  that 
controllers  who  worked  under  the  WRITING  condition  first,  reported  being  more  tired  under 
the  second  test  condition,  F(  1,6)  =  1.97,  p<.05. 

None  of  the  other  main  effects  or  interactions  were  found  significant  for  the  post-session 
questionnaire  variables,  p>.05. 

For  the  preliminary  questionnaire  variables,  partial  correlations  indicated  that  level  of  video- 
game  experience  was  positively  correlated  with  SA  under  the  NO  WRITING  condition  (+.84, 
p<.05),  see  table  3-1.  This  relationship  will  be  discussed  in  further  detail  in  a  subsequent 
section.  None  of  the  preliminary  questionnaire  variables  correlated  significantly  with  SCORE 
in  the  NO  WRITING  condition  and  none  of  these  variables  were  found  to  correlate  with  SA  or 
SCORE  in  the  WRITING  condition. 


3.1  ADDITIONAL  ANALYSES. 

The  data  were  analyzed  further  in  order  to  learn  more  about  the  control  strategies  and  flight  strip 
management  techniques  used  to  work  through  the  test  conditions  presented  in  the  current 
experiment.  Quantitative  measures  of  several  of  these  strategies  and  techniques  are  described 
below.  Although  many  of  the  differences  reported  were  not  found  to  be  significant,  they  suggest 
some  potential  trends  and  are,  therefore,  worthy  of  report. 
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TABLE  3-1.  CORRELATION  MATRIX  FOR  PRELIMINARY  QUESTIONNAIRE 
VARIABLES  AND  SA  IN  THE  NO  WRITING  CONDITION. 


SA 

YRS 

CO 

VI D 

VIS 

STR 

MO 

SA 

1.00 

.197 

.484 

.842 

-.008 

-.091 

-.231 

YRS 

.197 

1.00 

-.076 

-.075 

.243 

.460 

-.244 

CO 

.484 

-.076 

1.00 

.755 

.332 

-.206 

-.137 

V1D 

.842 

-.075 

.755 

1.00 

-.158 

-.141 

.052 

VIS 

-.008 

.243 

.332 

-.158 

1.00 

.201 

-.373 

STR 

-.091 

.460 

-.206 

-.141 

.201 

1.00 

.586 

MO 

-.231 

-.244 

-.137 

.052 

-.373 

.586 

1.00 

Note:  SA  =  situational  awareness,  YRS  =  years  of  ATC  experience,  CO=-computer  experience,  VID  =  video  game 
experience,  VIS  =  vision,  STR  =  stress,  MO  =  motivation. 


Further  analyses  revealed  that  there  was  a  significant  negative  correlation  between  the  number 
of  commands  issued  in  the  NO  WRITING  condition  and  SA,  (-.83,  p<.01,  df=7).  SA 
decreased  as  the  number  of  transmissions  made  increased.  This  correlation  did  not  reach 
significance  for  the  WRITING  condition,  (-.73,  p>  .05,  df=7).  This  result  suggests  that  it  was 
more  difficult  for  participants  to  maintain  critical  information  in  memory  the  more  commands 
they  issued,  particularly  when  they  were  unable  to  record  this  information  on  flight  strips. 
Controllers  in  this  experiment  wrote  an  average  of  only  47  percent  of  their  actions  on  their  flight 
strips.  They  wrote  altitude  changes  almost  exclusively,  as  they  are  required  on  the  job.  Altitude 
changes  were  recorded  significantly  more  often  than  turn/heading  changes  or  speed  changes, 
F(2,6)  =  15.30,  p<.001  (see  table  3-2). 


TABLE  3-2.  PROPORTION  OF  COMMANDS  WRITTEN  BY  TYPE. 


ALTITUDE 

TURN 

SPEED 

.886  (.312) 

.041  (.067) 

.313  (.513) 
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These  proportions  were  obtained  by  comparing  the  number  of  commands  written  by  the  number 
of  commands  issued  in  each  category.  The  high  proportion  of  altitude  changes  recorded  also 
reflects  the  fact  that  participants  wrote  proposed  altitude  changes  on  their  strips  that  they 
sometimes  did  not  carry  out. 

The  data  were  also  examined  to  determine  whether  different  types  of  commands  were  issued  in 
the  two  test  conditions.  It  might  be  expected  that  proportionately  fewer  altitude  changes  would 
be  made  under  the  condition  in  which  notes  could  not  be  kept  on  strips  since  participants  would 
not  have  their  usual  record  of  those  changes  and  would  presumably  be  less  familiar  with  ways 
to  remember  them.  The  use  of  flight  strips  for  recording  altitude  changes  over  other  status 
changes  is  meaningful.  Altitude  information  can  not  be  obtained  from  the  display  screen  as 
directly  as  tum/heading  changes,  for  example.  Those  changes  can  be  observed  by  looking  at 
the  direction  of  the  aircraft’s  icon  on  the  scope.  The  only  information  provided  on  the  scope 
about  an  altitude  change  is  the  up/down  (ascending/descending)  arrow  notation  located  next  to 
the  aircraft’s  current  altitude,  all  of  which  is  presented  below  the  call  sign.  The  controller 
cannot  determine  the  exact  altitude  to  which  he/she  has  sent  the  aircraft  unless  the  information 
is  remembered  or  available  on  the  flight  strip.  Investigating  whether  control  strategies  differed 
between  test  conditions  seemed  especially  worthwhile  since  one  controller  described  that,  in  the 
event  of  radar  failure  when  only  flight  strips  are  available,  altitude  separation  is  used  almost 
exclusively. 

The  proportion  of  command-types  issued  under  each  test  condition  was  examined.  The 
difference  between  the  overall  proportion  of  command  types  issued  under  each  test  condition  did 
not  reach  significance,  F(i,6)=3.92,  p>  .05.  The  TYPE  x  CONDITION  interaction  was  also 
not  significant  (p>  .05),  indicating  that  the  proportion  of  these  different  command  types  did  not 
differ  when  participants  had  the  opportunity  to  write  on  strips  than  when  they  did  not. 
Participants,  therefore,  did  not  issue  significantly  fewer  altitude  changes,  or  significantly  more 
tum/heading  changes,  when  they  were  unable  to  write  on  strips. 

Another  attempt  was  made  to  investigate  differences  between  the  two  test  conditions  by 
comparing  only  memory  for  the  last  command,  if  that  command  was  an  altitude  change.  Using 
the  reasoning  above,  it  should  presumably  be  more  difficult  for  controllers  to  remember  altitude 
changes  if  they  were  unable  to  write  on  strips.  Altitudes  were  the  last  commands  issued  in  only 
two  to  eight  occasions  in  each  test  condition.  Since  the  last  command  issued  at  each  test  interval 
varied  from  one  individual  to  another,  proportions  were  not  based  on  an  equal  number  of 
observations  across  either  individuals  or  test  conditions.  Using  the  information  that  was 
available,  it  was  found  that  the  mean  proportion  of  altitudes  recalled  in  the  NO  WRITING 
condition  was  .557  (SD  =  .334)  and  .729  (SD  =  .215)  in  the  WRITING  condition.  This 
difference,  however,  did  not  reach  significance,  F(l,7)  =  1.56,  p>.05. 

Other  aspects  of  the  participants’  flight  strip  management  techniques  were  investigated.  Since 
participants  used  flight  strips  that  were  placed  in  separate  holders  and  in  strip  bays,  they  were 
able  to  alter  strip  placement  by  rearranging  the  order  of  the  strips,  offsetting  ("cocking")  them 
left  or  right,  or  by  grouping  them.  One  possible  reason  why  no  differences  in  SA  were  found 
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between  the  test  conditions  may  have  been  due  to  the  use  of  different  strip  management 
techniques  in  each  condition.  These  different  techniques  may  have  been  used  as  memory  joggers 
when  writing  on  strips  was  not  allowed.  For  example,  a  strip  may  have  been  offset  to  indicate 
that  a  further  action  was  needed,  or  several  strips  may  have  been  grouped  together  to  indicate 
that  all  were  going  to  the  same  destination.  Participants  may  have  been  using  more  of  these 
techniques  in  the  NO  WRITING  condition  to  compensate  for  their  inability  to  keep  notes. 

To  determine  this,  the  video  tapes  of  the  strip  management  activities  were  analyzed.  All  non¬ 
writing  activities  were  sorted  into  categories  and  counted.  Non-writing  activities  included 
cocking  the  strips,  separating  them  into  groups,  rearranging  strip  placement,  bringing  strips  from 
the  pending  to  the  active  bay,  removing  strips  from  the  active  bay,  and  sliding  strips  in  the 
active  bay  into  better  view.  A  proportion  of  organizational  (grouping)  or  "cue-related" 
(offsetting)  activities  was  determined  from  the  total.  This  proportion  did  not  include  actions 
which  brought  strips  into  and  out  of  the  active  bay  (without  organizing  or  offsetting  them),  those 
which  involved  touching  but  not  moving  the  strip,  or  those  in  which  all  the  active  strips  were 
moved  to  a  more  visible  location  (i.e.,  slid  further  down  the  bay  and  into  view),  leaving  their 
relative  placement  unchanged. 

The  mean  proportion  of  flight  strip  management  activities  was  23.57  (SD=5.6)  in  the  NO 
WRITING  condition  and  20.29  (SD=6.6)  in  the  WRITING  condition.  The  small  difference 
observed  in  the  means,  however,  did  not  reach  significance,  t(6)  =  +  1.70,  p>.05.  From  the 
proportion  of  activities  included,  it  could  not  be  determined  that  more  "organizational"  or  "cue- 
related"  activities,  as  categorized  here,  were  used  in  the  NO  WRITING  condition. 

Finally,  one  very  compelling  result  of  this  experiment  was  observed.  Several  controllers  had 
commented  on  their  inability  to  remember  aircraft  call  signs  in  the  current  experiment, 
something  which  they  did  not  feel  was  a  problem  for  them  on  the  job.  It  is  assumed  that 
controllers  do  remember  call  signs  better  on  the  job,  but  such  information  was  not  available  in 
the  current  experiment.  Controllers  in  this  experiment  remembered  an  average  of  only  82 
percent  of  the  call  signs.  One  controller  remembered  just  20  percent.  Memory  for  call  signs 
was  also  examined  between  test  conditions,  under  the  assumption  that  it  may  have  been 
somewhat  easier  for  controllers  to  remember  call  signs  when  they  had  some  type  of  further 
interaction  with  the  aircraft  information.  Call  signs  were  recalled  79  percent  of  the  time  under 
the  NO  WRITING  condition  and  84  percent  of  the  time  under  the  WRITING  condition,  a 
difference  that  did  not  reach  significance,  p>  .05. 

The  observation  that  controllers  did  not  remember  aircraft  call  signs  as  well  as  they  report  they 
normally  do,  is  important.  Controllers  indicated  that  this  was  due  to  their  having  to  issue 
commands  via  the  keyboard,  rather  than  verbally.  The  fact  that  controllers  were  not  saying  the 
aircraft  call  signs  out  loud  prior  to  issuing  a  command  may  have  made  the  call  signs  less  salient 
in  memory.  The  articulatory  loop  notion  supports  this  view.  It  proposes  that  articulation  of 
information  is  valuable  in  assisting  recall.  When  articulation  of  material  is  prevented  or  when 
articulation  of  conflicting  material  is  required,  memory  deteriorates  (Baddeley,  Thompson,  and 
Buchanan,  1975).  On  a  broader  scale,  this  also  reflects  the  importance  of  active  encoding. 
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Physically  (here,  verbally)  interacting  with  material  enhances  its  ability  to  be  remembered. 
Without  such  interaction,  memory  has  been  observed  to  be  degraded.  The  observation  that  call 
signs  were  not  remembered  with  high  accuracy  in  this  experiment  should  be  taken  into 
consideration  and  investigated  further  before  any  changes  to  reduce  or  eliminate  verbal 
communication  are  made  to  the  ATC  system. 


3.2  OTHER  FACTORS  AFFECTING  PERFORMANCE. 


Information  from  the  preliminary  questionnaire  indicated  that  participants’  SA  in  the  NO 
WRITING  condition  was  correlated  with  level  of  reported  video-game  experience.  SA  was 
higher  for  participants  who  reported  a  higher  level  of  video-game  experience.  To  further 
analyze  this  data,  the  participants  were  divided  according  to  the  level  of  video-game  experience 
reported.  Those  reporting  less  than  the  average  value  (4)  were  designated  the  "LOW  VIDEO" 
group,  and  those  reporting  a  value  higher  than  4  were  designated  the  "HIGH  VIDEO"  group. 
SA  for  these  groups  is  presented  in  figure  3-2. 


Note.  Standard  deviations  above  data  points. 


FIGURE  3-2.  SA  SCORE  AS  A  FUNCTION  OF  VIDEO-GAME  EXPERIENCE 


An  ANOVA  indicated  that  neither  the  CONDITION  nor  the  GROUP  main  effects  were 
significant  (p>.05),  however,  the  GROUP  x  CONDITION  interaction  was  significant, 


20 


F(l,6)=9.32,  p<.05.  Planned  contrasts  revealed  that  the  HIGH  and  LOW  VIDEO  groups 
performed  equally  well  under  the  WRITING  condition,  F(l,6)  =  .007,  p>.05,  but  the  LOW 
VIDEO  group  performed  more  poorly  under  the  NO  WRITING  condition,  F(l,6)  =  8.62,  p<  .05. 
The  LOW  VIDEO  group’s  SA  improved  between  the  NO  WRITING  and  WRITING  conditions, 
F(  1,6)=  10.91,  p<  .05,  but  the  HIGH  VIDEO  group’s  SA  did  not  differ  between  test  conditions, 
F(l,6)  =  1.07,  p>  .05. 

This  result  indicates  that  those  participants  who  were  less  familiar  with  video  games  were  less 
able  to  maintain  SA  when  they  were  unable  to  use  a  written  memory  backup.  It  is  possible  that 
so  much  of  their  concentration  was  focused  on  manually  issuing  commands  and  receiving 
feedback  from  the  screen,  that  they  were  unable  to  devote  enough  attention  to  the  memory 
component  of  the  task  required.  Having  less  experience  in  these  areas  may  require  that  too 
much  attention  be  devoted  to  those  tasks  and  may  take  away  from  the  ability  to  keep  other 
aspects  of  the  situation  in  memory. 

SA  had  been  measured  by  combining  the  proportion  of  correct  responses  regarding  aircraft 
locations  and  the  proportion  of  correct  responses  regarding  last-issued  commands  into  a  single 
score.  The  question  remained  as  to  whether  differences  between  these  groups  of  controllers 
were  due  to  differences  between  one  of  these  measures  more  than  the  other.  Memory  for 
locations  and  memory  for  last-issued  commands  were  therefore  analyzed  separately. 

Mean  proportions  of  correct  responses  for  aircraft  locations  are  presented  in  Figure  3-3  for  each 
controller  group  and  each  test  condition. 


LOW  VIDEO  HIGH  VIDEO 


Note.  Standard  deviations  adjacent  to  data  points. 


FIGURE  3-3.  MEAN  PROPORTION  CORRECT:  LOCATIONS  OF  AIRCRAFT 
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Groups  did  not  differ  significantly  with  respect  to  the  proportion  of  correctly  located  aircraft, 
F(l,6)  =  .06,  p>.05.  Neither  were  there  significant  differences  found  as  a  function  of 
CONDITION,  F(l,6)  =  .68,  p>.05,  or  CONDITION  x  GROUP,  F(l,6)=2.49,  p>.05. 
Controllers  with  different  levels  of  video-game  experience  did  not  differ  on  their  ability  to 
correctly  indicate  locations  of  aircraft  when  the  display  was  not  visible. 

Each  group’s  mean  proportion  of  correct  responses  regarding  last-issued  commands  under  each 
condition  is  presented  in  figure  3-4.  Groups  did  differ  in  their  memory  for  last  commands 
issued,  F(  1,6) =6. 28,  p<  .05.  The  LOW  VIDEO  group’s  memory  was  poorer  than  the  HIGH 
VIDEO  group’s.  Contrasts  indicated  that  differences  between  these  groups  neared,  but  did  not 
reach  significance  under  the  NO  WRITING  condition,  F(l,6)=5. 16,  p>  .05.  Differences  as  a 
function  of  CONDITION  were  not  significant,  F(i,6)  =  .33,  p>  .05,  nor  were  differences  as  a 
function  of  CONDITION  x  GROUP,  F(1 ,6)  =  1. 13,  p>.05. 


LOW  VIDEO  HIGH  VIDEO 


Note.  Standard  deviations  adjacent  to  data  points 


FIGURE  3-4.  MEAN  PROPORTION  CORRECT:  LAST  COMMAND  ISSUED 


Memory  for  last-issued  commands  was  investigated  further  for  these  groups  to  determine 
whether  there  were  differences  in  the  types  of  information  that  each  group  was  able  to  recall. 
The  analysis  focused  on  determining  whether  controllers  in  the  LOW  VIDEO  group  were  less 
able  to  recall  altitudes  than  those  in  the  HIGH  VIDEO  group.  This  analysis  was  conducted 
because  controllers  typically  use  flight  strips  to  record  changes  in  altitude,  a  finding  that  was 
also  observed  in  this  experiment.  Since  controllers  in  the  LOW  VIDEO  group  appeared  more 
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dependent  on  strips,  they  were  expected  to  remember  fewer  altitude  changes  when  note-writing 
on  strips  was  not  allowed. 

The  proportion  of  correctly-remembered  altitude  changes  was  measured,  rather  than  the  actual 
number  of  altitude  changes,  since  the  actual  number  varied  for  each  individual  and  each  test 
condition.  The  last  command  issued  was  determined  solely  by  the  actions  of  the  controller  and 
so  the  experimenters  did  not  have  control  over  the  types  of  commands  that  would  be  issued  last 
to  each  aircraft  prior  to  pausing  the  scenario.  Mean  proportions  of  remembered  altitude  changes 
are  presented  in  figure  3-5. 


-h*“  LOW  VIDEO  — HIGH  VIDEO 


Note.  Standard  deviations  adjacent  to  data  points. 


FIGURE  3-5.  MEAN  PROPORTION  CORRECT:  LAST  ALTITUDE  CHANGE  ISSUED 


Groups  differed  in  terms  of  the  proportion  of  altitude  changes  that  they  correctly  remembered. 
The  LOW  VIDEO  group  remembered  a  significantly  lower  proportion  of  altitude  changes  than 
the  HIGH  VIDEO  group,  F(l,6)  =5.91,  p<.05.  These  controllers  differed  in  terms  of  their 
ability  to  recall  altitudes  in  the  NO  WRITING  condition  as  indicated  by  contrasts.  Controllers 
in  the  LOW  VIDEO  group  remembered  a  significantly  lower  proportion  of  altitude  changes  than 
did  those  in  the  HIGH  VIDEO  group  under  the  NO  WRITING  test  condition,  F(l,6)=8.23 
,p<  .05.  Differences  as  a  function  of  CONDITION  were  not  significant  overall,  F(l,6)=2.02, 
p>  .05,  nor  were  differences  as  a  function  of  GROUP  x  CONDITION,  F(l,6)=3.07,  p>  .05. 
For  the  LOW  VIDEO  group,  the  difference  between  the  proportion  of  altitude  changes 


23 


remembered  in  each  test  condition  neared,  but  did  not  reach  significance,  F(l,6)=5.04,  p>  .05, 
as  indicated  by  contrasts.  Controllers  in  this  group  were  somewhat  less  accurate  in  remembering 
altitude  changes  in  the  NO  WRITING  condition  than  they  were  when  they  were  able  to  keep 
notes  on  strips.  No  suggestion  of  such  a  trend  was  found  for  controllers  in  the  HIGH  VIDEO 
group. 


4,  DISCUSSION. 

The  results  indicated  that  there  were  no  differences  in  situational  awareness  or  control 
performance  between  test  conditions  when  level  of  video-game  experience  was  not  considered. 
This  result  suggests  that  writing  on  flight  strips  is  not  as  critical  to  ATC  performance  or  for 
keeping  the  "picture"  as  has  been  previously  suggested  (Hopkin,  1991).  However,  in  this 
experiment,  terminal  controllers  were  studied  exclusively.  Different  results  may  have  been 
found  for  en  route  controllers. 

While  note-writing  was  not  found  to  improve  controllers’  situational  awareness,  it  is  still  possible 
that  other  flight  strip  management  activities  may  have  been  valuable.  A  follow-up  experiment, 
comparing  situational  awareness  under  conditions  in  which  participants  can  and  cannot  rearrange 
strip  placement,  for  example,  would  be  needed  to  determine  the  importance  of  such  activities. 

There  are  several  alternative  explanations  for  the  results.  One  possible  explanation  is  that  the 
scenarios  used  in  this  experiment  may  not  have  required  the  use  of  flight  strips  in  order  for 
controllers  to  maintain  situational  awareness.  For  example,  controllers  may  have  been  able  to 
identify  patterns  in  the  traffic  which  they  used  to  guide  their  organization  of  aircraft  locations 
and  their  memory  for  the  action  steps  taken  with  them. 

Another  possible  explanation  is  that  controllers  thought  both  test  conditions  were  too  easy  to 
have  found  flight  strip  management  activities  like  note-writing  beneficial.  This  seems  unlikely, 
however,  given  that  controllers’  workload  reports  were  moderately  high,  averaging  7.3  on  a  12- 
point  scale,  suggesting  that  they  felt  at  least  somewhat  challenged  by  the  scenarios  provided. 

Another  explanation  may  lie  in  the  method  by  which  situational  awareness  was  measured. 
Participants  in  this  experiment  knew  in  advance  that  the  scenarios  would  be  stopped  at  various 
times  and  that  they  would  be  specifically  asked  about  the  last  commands  issued  and  the  locations 
of  the  aircraft.  Pausing  the  scenario  and  probing  directly  for  specific  information  could  have 
caused  controllers  to  focus  on  only  those  components  of  the  task  while  ignoring  others  and  may 
have  biased  them  to  use  different  memory  techniques  than  they  normally  would  on  the  job.  For 
example,  controllers  may  have  rehearsed  some  of  this  information  in  order  to  remember  it  for 
these  test  purposes,  but  may  not  typically  rehearse  such  information  when  working  with  real 
traffic.  Such  concerns  about  intrusive  methods  of  situational  awareness  assessment  have  been 
raised  previously  (Sarter  and  Woods,  1991). 
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Some  differences  in  controller  situational  awareness  may  have  occurred  in  the  present 
experiment,  but  the  measures  used  here  may  have  been  too  insensitive  to  detect  them.  Sample 
size  was  small  and  power  may  not  have  been  sufficient  to  detect  significant  differences.  Larger 
sample  sizes  along  with  less  intrusive  measures  of  situational  awareness  may  be  valuable  in 
determining  whether  controllers’  situational  awareness  differs  under  such  conditions. 

Another  explanation,  and  one  supported  by  the  work  conducted  here,  is  that  certain  individuals 
may  benefit  more  from  flight  strip  management  activities  than  others,  given  the  nature  of  the 
task  demands.  In  this  experiment,  controllers  who  reported  lower  levels  of  video-game 
experience  benefitted  from  keeping  notes  on  flight  strips.  Their  situational  awareness  improved 
under  this  condition.  Controllers  who  reported  higher  levels  of  video-game  experience 
maintained  situational  awareness  equally  between  both  test  conditions.  Additionally,  further 
analyses  indicated  that  different  aspects  of  situational  awareness  were  affected.  Groups  did  not 
differ  in  terms  of  their  memory  for  visual  information.  Memory  for  aircraft  locations  did  not 
differ  between  the  groups  or  between  the  test  conditions.  However,  groups  did  differ  with 
respect  to  memory  for  other,  non-visual  components  of  the  task.  Memory  for  commands, 
specifically  memory  for  altitude  changes,  was  lower  for  those  reporting  a  lower  level  of 
experience.  Differences  between  groups  were  most  prominent  for  the  NO  WRITING  condition. 
Not  being  able  to  write  on  flight  strips  was  apparently  more  detrimental  for  controllers  with 
lower  video-game  experience  than  for  those  reporting  a  higher  level  of  experience. 

These  results  suggest  that  individuals  who  are  not  experienced  interacting  with  equipment  in 
which  commands  are  issued  manually  and  feedback  is  received  visually  may  have  a  more 
difficult  time  maintaining  situational  awareness  without  a  memory  backup.  Note-writing  may 
be  helpful  to  these  individuals  because  it  may  provide  a  way  for  them  to  keep  track  of  critical 
information  when  much  of  their  attention  needs  to  be  devoted  to  the  technique  involved  in 
interacting  with  the  equipment.  Such  a  result  has  implications  for  the  performance  of  certain 
individuals  in  more  automated  systems  that  rely  on  this  form  of  interaction. 

Finally,  one  other  compelling  finding  of  the  current  experiment  was  that  controllers  indicated 
being  less  able  to  remember  aircraft  call  signs  under  these  conditions,  in  which  commands  were 
issued  via  the  keyboard,  than  they  are  on  the  job,  in  which  commands  are  issued  verbally. 
Future  work  will  need  to  identify  whether  this  phenomenon  holds  after  participants  have  had 
more  experience  with  the  equipment.  Nevertheless,  this  finding  also  has  ramifications  for  any 
system  that  will  involve  the  use  of  computer-entered  commands.  The  potential  exists  for 
controllers  to  have  difficulty  identifying  aircraft  appropriately. 


5.  RECOMMENDATIONS. 

The  results  obtained  in  this  experiment  suggest  that  certain  individuals  may  benefit  from  using 
flight  strip  management  techniques  as  a  memory  back-up  when  keyboard  entries  are  required. 
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Future  work  is  needed  to  determine  how  these  individuals  are  affected  by  continued  practice  with 
such  systems  and  whether  the  reliance  on  such  memory  aids  declines  with  increasing  experience. 

The  finding  that  different  individuals  benefitted  from  different  memory  strategies  under  various 
task  conditions  suggests  that  research  into  effective  memory  aids  may  need  to  be  pursued  from 
an  individualized,  controller-centered  perspective.  Programs  that  train  controllers  to  identify 
their  own  memory  limitations  and  requirements  and  which  provide  controllers  with  a  variety  of 
memory-aiding  strategies  that  can  be  implemented  on  a  situation-specific  basis,  may  prove  the 
most  useful  approach.  Controllers,  for  example,  can  be  trained  to  become  more  aware  of  factors 
that  influence  memory  and  performance  (e.g.,  stress,  fatigue).  They  can  be  instructed  on  how 
to  enhance  their  self-awareness  of  these  factors  and  on  how  to  determine  the  extent  to  which  they 
are  affected  by  them. 

Controllers  can  also  be  specifically  instructed  on  a  variety  of  general  memory-enhancing 
principles,  such  as  task  organization  and  standardization,  and  trained  on  how  to  develop  and 
implement  these  skills  as  needed.  Examples  can  be  provided  and  controllers  can  practice 
implementing  these  skills  using  realistic  scenarios  presented  on  high-fidelity  simulators. 
Emphasizing  the  development  of  individualized  memory  strategies  is  useful  in  that  the  basic 
skills  acquired  can  be  applied  to  a  variety  of  work  conditions.  They  would  therefore  be  useful 
to  controllers  over  the  long-term,  in  both  current  and  future  air  traffic  control  systems.  Such 
training  programs  need  to  be  developed  and  evaluated  since  they  provide  a  potentially  promising 
way  to  reduce  cognitive  load  and  improve  ATC  safety  and  efficiency. 


6.  IMPLICATIONS. 

There  are  several  important  implications  that  emerge  from  this  research.  They  are: 

a.  Controllers’  performance  with  certain  ATC  systems  may  depend  upon  the  amount  of 
prior  experience  they  have  had  with  complex  systems  like  video  games  which  require  rapid 
decision  making  and  skilled  eye-hand  coordination.  Controllers  with  more  extensive  experience 
using  such  interactive  devices  may  be  more  readily  able  to  perform  the  kinds  of  tasks  required 
here  without  having  to  rely  on  memory  aids.  In  this  experiment,  controllers  with  higher  video- 
game  experience  performed  equally  well  under  both  test  conditions. 

b.  Memory  lapses  and  effective  memory-aiding  techniques  may  differ  between  different 
subgroups  of  controllers  depending  on  their  knowledge  of  and  skill  level  with  various  systems. 
In  this  experiment,  only  those  controllers  reporting  lower  levels  of  video-game  experience 
improved  their  situational  awareness  (SA)  significantly  when  they  were  allowed  to  keep  notes 
on  flight  strips. 

c.  Results  of  research  conducted  with  low-fidelity  systems  can  not  be  directly  translated 
to  the  present  ATC  environment.  In  this  experiment,  the  simulator  used  required  keyboard 
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rather  than  verbal  communication  to  issue  commands  to  aircraft.  This  condition  made  the 
control  of  aircraft  very  different  from  that  used  in  the  real-world  ATC  system.  One  result  was 
that  controllers  reported  being  less  able  to  remember  aircraft  call  signs  in  this  experiment  than 
they  report  they  are  on  the  job.  Although  controllers  were  willing  and  able  to  use  the  low- 
fidelity  system,  testing  must  be  conducted  with  high-fidelity  simulators  and  eventually  in  the 
operational  environment  in  order  for  the  results  of  the  work  to  be  directly  pertinent  to  current 
job  conditions. 
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APPENDIX  A 


PILOT  STUDIES  WITH  NON-CONTROLLERS 


Pilot  Work  on  Effects  of  Flight  Strip  Use  on  Situational  Awareness: 
Non-Controller  Participants 


Two  groups  of  non-controllers  participated  in  pilot  work  to  obtain  preliminary  information  as  the  usefulness  of  flight 
strips  to  situational  awareness  (SA).  Practice,  training,  and  testing  were  conducted  using  methods  similar  to  those 
described  for  controllers.  The  results  for  each  of  these  groups  and  the  specific  differences  in  methodology  used 
to  conduct  this  work  are  indicated  below. 

The  first  group  of  non-controllers  tested  consisted  of  aviation  students  from  Mercer  County  Community  College 
(MCCC)  and  were  recruited  from  the  school’s  flight  club.  This  was  the  same  poo!  from  which  participants  had 
been  selected  for  a  prior  experiment  (Zingale,  Gromelski,  and  Stein,  1992).  Priority  was  given  to  those  students 
who  had  participated  in  the  earlier  experiment  in  order  to  take  advantage  of  their  experience  with  TRACON.  Data 
from  eight  participants  were  included  in  the  final  analyses,  three  of  whom  had  participated  previously.  All  students 
who  completed  the  experiment  received  4  hours  of  flight  time  as  an  incentive  for  participation  in  the  study. 

The  ages  of  the  participants  in  this  group  ranged  from  19  to  25  (mean=21.25,  SD=2.55).  They  had  completed 
froin  zero  to  two  semesters  in  the  aviation  program  (mean  =  .75,  SD=1.04)  and  from  zero  to  six  ground  courses 
(mean  =  2. 13,  SD  =  2.36)  prior  to  their  participation  in  this  experiment.  Their  flight  experience  varied  considerably, 
ranging  from  23  to  165  hours  of  actual  flight  time  (mean  =  73.13,  SD  =  47.50).  Three  of  these  participants  also 
reported  having  some  additional  aviation  experience  prior  to  attending  MCCC. 

Students  gave  themselves  fairly  low  ratings  in  terms  of  their  computer  experience  (mean  =  3. 87,  SD=1.89)  and 
somewhat  higher  ratings  in  terms  of  their  video-game  experience  (mean=6.12,  SD=2.30).  Reported  stress  levels 
varied  (mean  =  4.50,  SD  =  2.45).  Level  of  reported  vision  (mean  =  9.00,  SD=1.77)  and  health  (mean  =  9.63, 
SD  =  .52)  were  quite  high  as  was  motivational  level  (mean  =  2.00,  SD=  1.77).  Participants  also  indicated  that  they 
freely  volunteered  to  participate  in  the  study  (mean  =  9.50,  SD  =  .76). 

The  second  group  of  non-controller  participants  consisted  of  employees  of  the  Federal  Aviation  Administration 
(FAA)  Technical  Center.  This  group  had  worked  on  air  traffic  control  (ATC)-related  projects,  but  were  not 
controllers  by  profession  and  had  no  formal  training  in  air  traffic  control.  These  participants  were  recruited  through 
an  office  memo  which  briefly  described  the  study  and  requested  that  anyone  interested  in  volunteering  attend  an 
introductory  meeting.  A  total  of  seven  participants  from  this  group  were  tested.  This  group  consisted  mostly  of 
people  with  advanced  degrees  and/or  considerable  work  experience  in  technical  areas,  such  as  computer 
programming. 

Two  of  the  participants  from  this  group  had  prior  experience  with  TRACON  or  with  a  similar  computer  game. 
Overall,  they  rated  themselves  quite  high  in  terms  of  their  computer  experience  (mean  =  8.00,  SD=  1.73),  and  fairly 
low  in  terms  of  their  video-game  experience  (mean  =  4.00,  SD  =  2.08).  Level  of  reported  vision  was  fairly  low 
(mean=7.00,  SD=2.08).  Stress  levels  varied  (mean =4.71,  SD=2.06).  These  participants  reported  themselves 
in  very  good  health  (mean  =  9.57,  SD  =  .53)  and  indicated  that  they  freely  volunteered  to  participate  in  the  study 
(mean=9.57,  SD  =  .53).  Motivational  level  was  generally  high  but  variable  (mean=2.71,  SD  =  3.30). 

These  groups  were  each  tested  at  their  own  sites,  using  the  equipment  available  at  each  location.  The  aviation 
students  were  tested  first,  in  one  of  the  computer  laboratories  on  the  MCCC  campus.  At  this  location,  computers 
with  slower  processing  capabilities  (286)  were  available.  Technical  Center  employees  were  tested  at  the  FAA 
Technical  Center’s  Human  Factors  Laboratory.  Here,  faster  computers  (486)  were  available.  Since  486  computers 
run  TRACON  at  a  noticeably  faster  rate  than  286’s,  this  necessarily  changed  the  nature  of  the  test  conditions 
between  these  groups. 

Additionally,  the  Technical  Center  employees  were  tested  using  the  updated  version  of  TRACON  II,  which  allows 
for  the  “exit  after  crash"  option  to  be  turned  off  so  that  the  scenario  can  be  run  to  completion.  The  TRACON 
version  used  with  the  aviation  students  did  not  allow  this  option.  With  this  version,  the  scenarios  end  following  a 


crash,  and  subsequent  data  can  not  be  collected.  As  a  result,  some  pieces  of  information  for  the  student  participants 
were  not  available.  Crashes  occurred  for  two  of  the  students  during  testing,  thus  eliminating  the  ability  to  obtain 
total  session  scores.  Given  the  different  circumstances  under  which  the  two  groups  of  non-controllers  were  tested, 
their  data  are  reported  separately. 

Participants  from  Mercer  College  had  more  flexibility  with  their  schedules  and  were  able  to  spend  more  time  in 
distributed  training  sessions.  The  participants  from  this  group  who  had  participated  in  the  prior  study,  completed 
at  least  two  training  sessions  of  1  to  1-1/4  hours  each  to  become  reacquainted  with  TRACON  and  to  become 
familiar  with  the  experimental  procedure.  New  students  were  provided  with  more  extensive  training  and  practice 
before  being  tested  under  the  experimental  conditions.  They  were  given  at  least  four  sessions  of  training.  In 
general,  this  group  was  observed  to  display  little  computer  knowledge  and  typing  experience.  They  required 
considerable  time  to  become  accustomed  to  entering  control  commands  via  the  keyboard.  Two  participants, 
originally  included  in  this  group,  were  eliminated  prior  to  testing  because  they  failed  to  attend  the  appropriate 
number  of  training  sessions  and  their  performance  was  markedly  lower  than  the  others.  (One  other  participant,  who 
did  participate  in  testing,  was  subsequently  eliminated  since  he  did  not  carry  out  one  of  the  test  conditions  as 
instructed.) 

Non-controller  Technical  Center  employees  were  available  at  various  times  throughout  a  one-week  time  block.  The 
seven  participants  in  this  group  were  scheduled  to  maximize  the  amount  of  training  that  could  be  conducted 
simultaneously  with  the  two  computers  available  in  the  testing  lab.  This  group  completed  at  least  three  training 
sessions  of  approximately  1  to  1-1/4  hours  each.  These  sessions  were  distributed  over  2  days.  Training  proceeded 
smoothly,  since  participants  in  this  group  were  computer  literate  and  had  good  typing  skills. 

Both  participant  groups  used  paper  flight  strips  that  were  presented  sequentially  on  8-1/2  x  11  inch  sheets  of  paper. 
Strip  placement,  therefore,  could  not  be  altered.  Prior  to  the  start  of  the  practice  and  test  sessions,  these  participants 
completed  questionnaires,  similar  to  those  completed  by  controllers,  to  identify  factors  that  may  have  additionally 
influenced  performance  (e.g.,  aviation  experience,  TRACON  experience,  stress  level). 

RESULTS. 

AVIATION  STUDENTS. 

The  aviation  students’  SA  differed  significantly  between  the  WRITING  and  NO  WRITING  conditions, 
F(  1,6)=  15.40,  p<  .01,  (see  figure  A-l).  This  group’s  SA  improved  when  they  were  allowed  to  write  notes  on  the 
strips. 
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FIGURE  A-l.  MEAN  SA  SC:.T>C  '  OR  STUDENTS  AS  A  FUNCTION  OF  CONDITION  AND  TEST  ORDER. 
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The  effect  of  ORDER  was  not  significant,  F(l,6)  =  .03,  p>.05,  nor  was  the  interaction  of  ORDER  x  CONDITION, 
F(l,6)=5.45,  .05<p<.10.  Performance  SCORES  for  this  group  also  did  not  differ  significantly  as  a  function  of 
CONDITION,  nor  as  a  function  of  ORDER  or  the  ORDER  x  CONDITION  interaction  (p>  .05). 

With  respect  to  the  post-session  questionnaire  variables,  this  group  indicated  that  flight  strips  were  significantly  more 
helpful  in  getting  them  to  remember  information  (MEMORY)  about  aircraft  in  the  WRITING  condition, 
F(l,6)  =  8.87,  p  <  .05.  Neither  the  effect  of  ORDER  nor  the  ORDER  x  CONDITION  interaction  were  significant, 
p>.05. 

For  other  variables,  only  the  secondary  main  effect,  ORDER,  and  the  ORDER  x  CONDITION  interaction  were 
found  to  be  significant.  A  significant  effect  of  ORDER  indicates  that  the  two  subgroups  of  participants  differed 
from  one  another  on  the  measure  tested.  Since  participants  were  randomly  selected  to  work  under  one  or  the  other 
condition  first,  this  result  was  not  expected  and  it  is  not  obvious  why  it  should  occur.  One  possibility  is  that  having 
worked  under  one  condition  first  influenced  the  way  in  which  each  group  managed  the  second.  A  significant 
ORDER  x  CONDITION  interaction  indicates  more  directly  that  the  measure  was  influenced  by  the  sequence  in 
which  the  conditions  were  conducted.  A  significant  interaction  may  indicate  a  practice  effect,  for  example,  if 
measures  for  each  group’s  first  test  condition  differed  from  their  second. 

For  students,  the  effect  of  ORDER  was  significant  for  the  CONTROL  variable,  F(l,6)  =  9.89,  p<.05.  Students 
who  worked  under  the  WRITING  condition  first,  reported  that  flight  strips  were  less  useful  in  helping  them  control 
traffic  than  did  those  who  worked  under  the  NO  WRITING  condition  first. 

The  effect  of  ORDER  was  also  found  significant  for  the  FATIGUE  variable,  F(l,6)=6.53,  p<  .05.  Students  who 
worked  under  the  WRITING  condition  first,  reported  a  lower  level  of  fatigue  overall.  None  of  the  other  post¬ 
session  questionnaire  variables  differed  significantly  as  a  function  of  CONDITION,  ORDER,  or  their  interaction 
(p  >  .05)  for  the  STUDENT  group. 

As  far  as  the  preliminary  questionnaire  variables  were  concerned,  two  variables,  level  of  reported  stress  and  level 
of  reported  vision,  were  significantly  correlated  with  SA  in  the  NO  WRITING  condition.  Partial  correlations 
indicated  an  inverse  relationship  between  SA  and  level  of  reported  stress  (-.49,  p<.05)  and  between  SA  and  level 
of  reported  vision  (-.78,  p<.05).  Improved  situational  awareness  was  associated  with  lower  levels  of  stress,  a 
result  also  obtained  in  the  prior  experiment.  Participants  who  indicated  lower  levels  of  stress  were  those  who  had 
displayed  better  performance.  Improved  SA  was  also  associated  with  lower  levels  of  reported  vision  under  this 
condition.  This  relationship  is  counterintuitive  and  it  is  not  obvious  what  this  result  may  mean.  None  of  the 
preliminary  questionnaire  variables  correlated  significantly  with  performance  SCORE  in  this  condition. 

As  for  the  WRITING  condition,  none  of  the  aviation  student’s  preliminary  questionnaire  variables  were  found  to 
correlate  with  S  A  (p  >  .05).  However,  number  of  flight  hours  was  found  to  correlate  significantly  with  performance 
SCORE  (+ .97,  p  <  .001).  The  greater  the  amount  of  flight  experience,  the  higher  the  performance  level  achieved 
under  this  test  condition. 


TECHNICAL  CENTER  EMPLOYEES. 

The  trends  in  the  results  obtained  for  this  group  appeared  very  different  from  those  observed  for  the  students  (see 
figure  A-2).  SA  decreased  significantly  for  this  group  under  the  WRITING  condition,  F(l,5)  =  7.85,  p<.05. 
Neither  the  effect  of  ORDER,  F(l,5)  =  .05,  p>.05,  nor  the  interaction  of  ORDER  x  CONDITION,  F(l,5)=1.80, 
p>  .05,  were  significant.  Performance  scores  did  not  differ  significantly  as  a  function  of  CONDITION,  ORDER, 
or  the  ORDER  x  CONDITION  interaction,  p>.05. 
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Nole.  Standard  deviations  above  each  data  point. 


FIGURE  A-2.  MEAN  SITUATIONAL  AWARENESS  SCORE  FOR  EMPLOYEES  AS  A  FUNCTION  OF 

CONDITION  AND  TEST  ORDER. 

For  this  group,  WORKLOAD  differed  significantly  between  the  test  conditions,  F(l,5)=  12.76,p<.05.  Participants 
rated  their  workload  level  higher  in  the  WRITING  condition  than  the  NO  WRITING  condition.  The  main  effect 
of  ORDER  was  not  significant,  F(l,5)=  1.01,  p>  .05.  Nor  was  the  ORDER  x  CONDITION  interaction  significant, 
F(l,5)=5.90,  p>.05. 

Performance  assessment  also  differed  significantly  between  the  test  conditions,  F(l,5)=7. 10,  p<  .05.  Overall,  these 
participants  gave  themselves  lower  ratings  on  their  performance  in  the  WRITING  condition  than  the  NO  WRITING 
condition.  The  main  effect  of  ORDER  was  not  significant,  F(l,5)  =  .43,  p>.05.  However,  the  interaction  of 
ORDER  x  CONDITION  was  significant,  F(1 ,5)=  12. 14,  p  <  .05.  Planned  contrasts  indicated  that  participants  who 
worked  unaer  the  WRITING  condition  first,  reported  a  lower  performance  level  in  the  WRITING  condition  than 
did  those  who  first  worked  under  the  NO  WRITING  condition,  F(l,5)=22.06,  p<.01.  None  of  the  other  main 
effects  or  their  interactions  vere  significant  for  this  group,  p>  .05. 

As  for  the  preliminary  questionnaire  variables,  partial  correlations  indicated  that  stress  was  inversely  related  to  SA 
(-.78,  p<.05)  under  the  NO  WRITING  condition,  as  had  been  found  for  the  student  group.  Higher  SA  was 
associated  with  lower  levels  of  reported  stress.  In  addition,  motivational  level  correlated  significantly  with  this 
group’s  performance  SCORE  (-.91,  p<.01).  Participants  who  indicated  a  higher  level  of  motivation  performed 
better  that  those  who  reported  lower  levels.  (A  negative  correlation  was  found  because  this  item  was  worded  in 
such  a  way  that  a  low  value  indicated  high  motivation,  and  a  high  value  indicated  low  motivation.) 

For  the  employees,  none  of  the  preliminary  questionnaire  variables  was  found  to  correlate  significantly  with  SA  or 
SCORE  under  the  WRITING  condition. 


A -4 


DISCUSSION. 


The  aviation  students’  situational  awareness  improved  when  they  were  allowed  to  use  flight  strips  to  record  notes. 
Their  self-assessment  ratings,  as  indicated  on  the  post-session  questionnaire,  also  suggested  that  they  found  writing 
on  flight  strips  valuable  for  helping  them  remember  information  about  aircraft. 

The  Technical  Center  employees  demonstrated  a  lower  level  of  situational  awareness  when  they  were  required  to 
use  strips  for  note-writing.  Their  subjective  reports  also  indicated  that  they  had  a  very  difficult  time  working  under 
this  condition.  Their  self-assessment  ratings  were  lower,  and  their  workload  levels  were  higher,  when  they  were 
required  to  write  notes.  This  group  seemed  overwhelmed  by  having  to  control  traffic  and  keep  notes  on  strips  at 
the  same  time. 

The  trends  in  the  results  differed  between  these  two  non-controller  groups.  It  is  likely  that  differences  between  their 
practice  schedules  and  the  testing  equipment  used  changed  the  nature  of  the  task.  One  interpretation  of  the 
differences  in  results  is  that  note-writing  on  strips  is  helpful  to  novices  once  they  have  had  the  opportunity  to 
effectively  incorporate  it  into  their  operating  procedure.  The  aviation  students,  who  had  their  practice  and  training 
sessions  distributed  over  the  course  of  several  days,  may  have  had  a  better  opportunity  to  integrate  flight  strip 
management  with  control  tasks  than  the  employee  group.  Students  may  have  been  able  to  figure  out  how  to  use 
flight  strips  to  their  advantage,  while  the  employee  group  may  not  have  had  the  opportunity  to  combine  both  tasks 
effectively  and  found  the  additional  writing  task  burdensome  and  detrimental  to  the  primary  job  of  controlling 
aircraft. 
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TRAINING  MATERIALS 
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TRAINING  MANUAL  FOR  TESTING 


COMMONLY  USED  TERMS,  ACRONYMNS  AND  ABBREVIATIONS  ON  FLIGHT  PROGRESS 
STRIPS . 


What  is  a  "Flight  Progress  Strip?" 


A  flight  progress  strip  is  a  written  record  of  an  aircrafts 
performance  as  it  either  takes-off,  lands  or  overflies  the  airspace 
controlled  by  the  controller  ( YOU1 .  A  typical  flight  plan  follows: 


■ 

00 

7o7 

sss 

of 

loo 

/HDDS  I/MS' 

Dou/a/£  Lfrx 

WHAT  DOES  THIS  MEAN????? 


SK234  — -Scandinavian  Airlines  flight  number  234 

707 — - Type  of  aircraft.  A  Boeing  707 

250k - Airspeed  250  knots 

0001— —Time  of  arrival  into  problem 

100 ' - Altitude  10,000  feet. (The  last  two  zeros  are 

always  dropped  in  a  flight  plan. 

MIDDS—  An  intersection  of  two  routes  used  by  pilots 
•  when  they  navigate. 

V165 - -  A  highway  in  the  sky.  Victor  165 

DOWNE -  Another  intersection. 

LAX  Twr—  The  airport  of  intended  landing,  Los  Angeles. 

BEHDINg  Flight  progress  strips  or  flight  plans  that  are 

inactive.  The  "pending"  file  will  appear  about  five 
minutes  prior  to  the  flights  needing  control  action. 
(PENDING  FILES  ARE  BLUE  ) 

ACTIVE  Flight  progress  strips  or  flight  plans  that  are  active 
and  under  your  control. 

(ACTIVE  FILES  ARE  GREEN) 


SELECTED  ,  Flight  progress  strip  that  you  are  currently  issuing 
instructions  to. 

(SELECTED  FILES  ARE  BLACK) 


IHR  (Tower)  Flight  plans  indicating  "TWR"  are  aircraft  destined  to 
land  at  the  airport  or  "twr"  specified.  Those  airports 

are:  LAX - Los  Angeles 

VNY - Van  Nuys 

LGB - Long  Beach 

TOA — - — Torrance 
SMO— - Santa  Monica 
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Tus# 


CTR  (Center)  Flight  plans  indicating  "CTR"  are  aircraft  overflying 
the  Los  Angeles  airspace  enroute  to  another  "ctr" 
sector.  All  identifiers  for  "ctr"  control  have  five 
(5)  letter  characters,  e.g.  MIDDS,  HASS A 


T/0.  (Take-off)  Flight  plans  indicating  "T/0"  are  aircraft  taking 

off  from  one  of  the  airports  indicated  under 
"twr".  These  aircraft  are  going  from  a  "twr"  to 
"ctr"  environment. 

*  (Asterisk)  Flight  plans  having  an  in  front  of  the 

routing  indicates  that  control  action  on  that 
flight  plan  is  needed. 


CONTROL  ACTIONS  NECESSARY  TO  SAFELY  MANAGE  TRAFFIC  IN  THE  LOS 
ANGELES  SECTOR 


1) .  OVERFLIGHTS  Route  of  flights  that  start  with  a  five  letter 
identifiers  such  as  MIDDS  and  end  with  "Ctr".  You  must  take  a  hand- 
off  from  the  center  controller  when  the  aircraft  flashes  or  blinks 
at  you,  monitor  the  aircrafts  flight  through  your  airspace, 
protecting  it  from  other  flights  at  the  same  altitude  and  initiate 
a  hand-off  to  another  center  controller  when  the  flight  is  five  rsi 
miles  or  less  from  the  last  five  letter  identifier  or  sector 
boundary . 


RULES; 


1.  Take  hand-offs  as  soon  as  possible 

2.  Keep  other  aircraft  at  the  same  altitude  at  least  3 
miles  away  from  each  other. 

3.  If  aircraft  are  less  than  3  miles  from  each  other,  you 
must  have  at  least  1,000  ft.  separation  between 
aircraft. 

4.  Make  final  hand-off  when  the  flight  is  five  miles  or 
less  from  the  sector  boundary. 


2 ) .  DEPARTURES  Route  of  flights  that  start  with  T/O  and  end  with 
"Ctr".  These  aircraft  are  on  the  ground  (pending)  at  the  various 
airports  waiting  for  you  to  release  them  (activate  the  flight 
plan).  Departure  aircraft  need  to  be  released,  separated  from  other 
active  traffic,  monitored  to  a  point  five  miles  from  the  sector 
boundary  (last  five  letter  identifier)  and  then  handed-off  to  the 
next  center  controller. 
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RULES; 


1.  Release  aircraft  as  soon  as  possible. 

2.  Make  sure  you  have  at  least  3  miles  vertical  or 
1,000ft.  horizontal  separation  from  all  other  traffic 
m  your  sector. 

3.  Insure  that  aircraft  are  at  the  flight  planned  altitude 
prior  to  making  your  hand-off. 

4.  Insure  that  aircraft  are  going  to  the  proper  exit  fix. 

5.  Make  final  hand-off  when  the  flight  is  five  miles  or’ 
less  from  the  sector  boundary. 


3 ) ■ -ARRIVALS  Route  of  flights  that  end  with  "Twr".  These  aircraft 
are  the  most  difficult  to  control  since  you  must  take  a  hand-off 
;5°?wthe  center  controller,  radar  vector  or  maneuver  the  aircraft 
to  the  appropriate  final  approach  fix,  descend  the  aircraft  to  the 

ST2^r*.?lt;L^Ude'  *turn  the  aircraft  on  the  final  approach  course  and 
clear  the  aircraft  to  contact  the  tower. 


RULES ; 


1. 

2. 


3. 

4. 


5. 


6. 


Take  hand-offs  as  soon  as  possible 

Make  sure  you  have  at  least  3  miles  vertical  or  1000  ft. 
horizontal  separation  from  all  other  aircraft  in  vour 
sector . 


Descend  aircraft  to  the  proper  final  descent  altitude. 
Radar  vector  or  maneuver  the  aircraft  towards  the  final 
approach  course. 


Turn  aircraft  on  to  the  final  approach  course  outside 
the  final  approach  fix  (F.A.F.)  on  a  heading  no  greater 
than  thirty  degrees  from  the  final  approach  heading  as 
indicated  on  the  airports  chart. 

Make  final  hand-off  to  the  tower  prior  to  the  F.A.F. 


EW-RQUTES  Route  of  flights  that  start  with  T/O  and  end 

tothta^ro,VTh^  firJfra5t  are  on  th.®  ground  at  one  airport,  waiting 
ff  rd  land  at  another  airport  in  your  sector.  You  must 
?!L,a+u1V? te  the  flight  as  indicated  under  DEPARTURES,  and 
ARRIVALS  the  instructlons  Pertaining  to  vectoring  as  listed  in 


RULES ; 


1.  Release  the  aircraft  as  soon  as  possible. 

2.  Follow  instructions  under  ARRIVALS  listed  above. 
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COMMONLY  USED  TERMINOLOGY 


Request  Vector - Aircraft  is  requesting  assistance  to  the  airport 

or  to  a  navigational  fix. 

Request  descent -  Aircraft  is  getting  close  to  the  airport  of 

intended  landing  without  having  had  a 
clearance  to  descend  to  the  final  approach 
altitude. 

Request  release -  Tower  controller  is  asking  permission  for  a 

flight  at  his/her  airport  to  fly  into  your 
sector  under  instrument  flight  rules. 

Missed  approach -  Aircraft  that  you  cleared  for  an  approach  at 

one  of  your  airports  cannot  make  a  safe  landing 
due  to  being  either  too  high  ,  too  close  to  the 
airport,  too  far  away  from  the  final  approach 
course,  or  being  at  the  wrong  altitude. 

Not  on  my  scope  yet - Center  controller  reminding  you  that  you  are 

handing  the  aircraft  off  outside  the  5  mile 
parameter  recognized  by  the  game. 


HOW  TO  COMMUNICATE  TO  AIRCRAFT  AND  OTHER  CONTROLLERS 


In  order  to  communicate  to  aircraft  or  other  controllers,  you  must 
take  three  specific  steps. 

STEP  ONE -  SELECT  AN  AIRCRAFT^ 

When  an  aircraft,  control  tower  or  center  controller  wants  you  to 
assume  control  of  an  aircraft  (or  select  the  aircraft),  you'll  hear 
the  request,  see  the  aircraft  blinking  at  you  and  see  the  request 
written  on  your  PC  at  the  bottom  of  your  screen.  (Pink  area)  You 
can  select  the  aircraft  by  either  of  the  following  means. 

a)  Scroll  ARROW  UP  or  ARROW  DOWN  to  HIGHLIGHT  THE  AIRCRAFT,  THEN 
PRESS  THE  ENTER  KEY. 

b)  TYPE  THE  AIRCRAFT  IDENTIFICATION  When  the  App/Dep.  prompt 
appears  in  the  pink  area,  THEN  PRESS  THE  ENTER  KEY. 


You  now  have  selected  the  aircraft  that  you  viah  to  control. 
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S1EP  TMQ 


ISSUE  A  COMMAND  INSTRUCTION. 


Issue  to  your  selected  aircraft  the  appropriate  command 
instructions  as  listed  in  the  AIRCRAFT  SELECTED  COLUMN  below. 


(KEYBOARD  ENTRIES) 


AIRCRAFT  SELECTED  NO  AIRCRAFT  SELECTED 


ARROW  UP - Climb  and  maintain  _  Scroll  up 

ARROW  DOWN - Descend  and  maintain _  Scroll  down 

ARROW  LEFT - Turn  left _  *************** 

ARROW  RIGHT - Turn  right _  *************** 

BACKSPACE - Disregard  previous  command  Cancel  last  entry 

SPEED - Change  speed  to  _  ***************** 

DIRECT  TO - Cleared  direct  to  _  ***************** 

SAY  HEADING — --Say  heading  and  airspeed  ***************** 

RESUME  NORMAL- -Resume  speed  and  own  navigation  ***************** 
HAND-OFF— -—Hand-off  to  CTR  or  TWR  ***************** 

HOLD  AT  ———Hold  at  (designated  fix)  ***************** 

ENTER - Release  traffic  on  ground  Select  aircraft 

Take  hand-off  from  center  (most  important) 

SEMI-COLON  (;)-To  issue  multiple  commands  ***************** 

e . g .  Command  ( ; )  Command 

PLUS  ( + ) — — — — — — — Zoom  in  ***************** 

MINUS  ( — ) — — — — — — Zoom  out  ***************** 

SLANT  ( / )  ——————Move  aircraft  leader  to-——  ***************** 


STEP  THREE - Define  the  specific  parameters  such  as  altitude  or 

heading  using  the  numbers  functions  on  the  left  keyboard. 

ALTITUDE - The  last  two  digits  of  the  altitude  are  always  omitted. 

e.g.  19  means  1900ft. ,  120  means  12,000ft. 

HEADING - To  turn  to  a  specific  number  of  degrees  use  two  digits. 

e.g.  20  means  alter  heading  twenty  degrees. 

To  turn  to  a  specific  heading  use  three  digits. 

e.g.  020  means  heading  Zero  Two  Zero  degrees. 


HfiW  TO  TAKE  A  RADAR  HAND-OFF  AND  RELEASE  A  DEPARTURE  AIRCRAFT. 

This  process  only  requires  two  steps. 

Step  One - Select  the  aircraft  and  hit  the  ENTER  KEY 

Step  tWQ - Depress  the  ENTER  KEY  for  the  second  time. 
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OTHER  INFORMATION  KEYBOARD  ENTRIES. 

These  entries  are  provided  for  your  information  and  are  usefux  to 
obtain  additional  information. 


ALT+F -  To  gain  access  to  the  TRACON  menu. 

ARROWS -  To  move  left  or  right,  up  or  down  in  the  menu. 

TAB -  To  move  the  cursor  into  a  different  field. 

ESC - To  cancel  last  entry. 

CTRL+F -  To  show  the  aircrafts  flight  plan  route. 

CTRL+T -  To  show  the  Airport  information. 

CTRL+A -  To  show  aircraft  performance  characteristics. 


TO  ACTIVATE  ANY  INFORMATION  REQUESTS  YOU  MUST  FOLLOW-UP  BY 
DEPRESSING  THE  ENTER  KEY. 


HOW  TO  MAXIMIZE  YOUR  SCORING. 

1)  Take  hand-off's  as  soon  as  possible. 

2)  Keep  all  aircraft  at  the  same  altitude  at  least  3  miles  or  1,000 
feet  away  from  each  other. 

3)  Turn  aircraft  on  to  the  final  approach  course  outside  of  the 
approach  gate  or  course  indicator. 

4)  Hand-off  aircraft  no  sooner  than  5  miles  from  the  sector 
boundary. 

5)  Don't  turn  or  maneuver  aircraft  unnecessarily . 

6)  Don't  forget  to  make  hand-off's  to  the  next  center  controller. 

7)  Turn  aircraft  onto  the  approach  course  at  the  proper  altitude 
and  at  a  heading  that  does  not  differ  by  more  than  thirty 
degrees  from  the  heading  on  your  chart. 

8)  Release  departure  aircraft  as  soon  as  possible. 

9)  Try  to  issue  multiple  commands  using  the  (;)  to  allow  you  to 
control  the  frequency. 

10)  DON'T  AIM  TN3  AIRCRAFT  AT  BACH  OTHER.  A  COLLISION  IS  AN 
AUTOMATIC  EXIT  FROM  THE  TEST. 


This  testing  material  will  be  fully  explained  to  you  by  your 
instructor  who  was  a  qualified  Air  Traffic  Controller.  Feel  free  to 
ask  any  questions  about  the  information  hand-out,  Air  Traffic 
Control  in  general,  or  memory  strategies  we  hope  to  teach  you. 

Remember,  we  expect  you  to  make  mistakes.  We  want  you  to  have 
"FUN" .  Try  to  do  your  best  but  don't  worry  if  you  get  behind  or 
can't  remember  everything.  You  are  not  expected  to  become  Air 
Traffic  controller  after  this  experiment. 

Thanks  again  for  your  volunteer  participation. 

The  staff  at  PERI. 


B-7 


APPENDIX  C 


FLIGHT  STRIPS  AND  RESPONSE  FORMS 
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TRACON  II  Scoring  Parameters 


successful  overflight 

+  500 

successful  departure  ' 

+  600 

successful  arrival 

+  800 

any  command 

-  10 

pause 

-  10000 

wrong  speed  at  handoff 

-  100 

wrong  altitude  at  handoff 

-  250 

missed  handoff 

-  500 

missed  approach 

-  250 

lost  aircraft  off  radar 

-  1500 

separation  error  (<3mile) 

-  1000 

separation  erros  (cimile) 

-  5000 

crash 

0 

weather 

-  5 

pilot  request  (success/ fail) 

+/-  30 

emergency  (success/fail) 

+/-  500 

Point  values  added  or  subtracted  from  score  for  each 
command/situation/error.  (Wesson  International,  1990,  p.  8-12) 
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APPENDIX  D 


QUESTIONNAIRES  AND  SCORING  SHEETS 


Fall  1992  -  Tech  Center 


PRELIMINARY  QUESTIONNAIRE 


The  purpose  of  this  questionnaire  is  to  find  out  something  about  your  background  and  current 
feelings  about  this  project  in  order  to  better  understand  your  performance  during  the  course  of 
the  study.  All  information  will  be  coded.  Your  name  will  not  be  linked  to  the  answers  you 
provide. 


Your  current  position: 


Years  in  ATC: 


Years  in  terminal  environment: 


Years  in  en-route  environment: 


Total  facilities  at  which  you  have  worked: _ 

Do  you  typically  use  strip  management  techniques  (marking,  <  tc.)  on  the  job? 
yes  no 

If  yes,  briefly  describe  what  kinds  of  techniques  you  use. 


Have  you  ever  used  TRACON  II? 
yes  no 

If  yes,  rate  your  level  of  proficiency 

123456789  10 

low  high 

D-l 


Using  the  scale  provided: 

Rate  your  level  of  computer  experience. 

123456789  10 

none  extensive 


Rate  your  level  of  video  game  experience. 

123456789  10 

none  extensive 


Rate  your  current  vision. 

123456789  10 

poor  excellent 


Do  you  have  a  color  vision  deficiency?  yes  no 
If  so,  what  is  it?  _ 


I  freely  volunteered  to  participate  in  this  project. 

123456789  10 

strongly  strongly 

disagree  agree 


I  am  currently  in  good  health. 


1  2  3  4  5 

strongly 
disagree 


6  7  8  9  10 

strongly 
agree 


D-2 


During  the  last  several  months,  I  have  been  experiencing  a  relatively  high 
level  of  stress. 

123456789  10 

strongly  strongly 

disagree  agree 


I  am  not  very  motivated  to  participate  in  this  study. 

123456789  10 

strongly  strongly 

disagree  agree 


Identify  the  reason(s)  why  you  decided  to  participate  in  these  experiments. 


CONTROL  INSTRUCTIONS  ISSUED 


AIRCRA7T 


pczn 


TURNS/  DIRECT 

HBADINOi  TO  I 


RAND  CLIMBS/  SPEED  CLEAR  .  SOLD  ERROR 
CPFSj  DESCENT 8|  RSSTRNS  APPRO  I  I 


NtB«l 


Amn 


nsiot 


NPH-5X 


NW^cn 


ColL 


CONTROL  INSTRUCTIONS  ISSUED 


DIRECT  HXND  CLIMBS/  SPEED  CL ERR  .  SOLD  ERROR 

TO  I  OPPSi  DESCENT S|  RESTRMS.  APPRO  I  I 


PERFORMANCE  REVIEW 


Aircraft  handled: 
Total  time  (minutes): 
Separation  conflicts: 
Handoff  errors: 


/_ 

/ 


YOUR  SCORE: 


Missed  approaches: 

Pilot  requests  granted/total: 
Emergencies  landed/crashed: 
MAXIMUM  POSSIBLE  SCORE: 


Choose  the  one  number  below  which  best  describes  how  hard  you  were  working  during  this 
period: 


DESCRIPTION  OF  WORKLOAD  CATEGORY 

RATING 

(CIRCLE 

ONE) 

VERY  LOW  WORKLOAD 

1 

All  tasks  were  accomplished  quickly  and  easily 

2 

3 

MODERATE  WORKLOAD 

4 

The  chances  for  errors  or  omissions  were  low 

5 

6 

I  RELATIVELY  HIGH  WORKLOAD 

7 

1  The  chances  for  some  errors  or  omissions  were 

8 

relatively  high 

9 

VERY  HIGH  WORKLOAD 

10 

It  was  barely  possible  to  accomplish  all  tasks 

11 

properly 

12 
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CONTROLLER  POST-SESSION  QUESTIONNAIRE 


PLEASE  COMPLETE  THE  FOLLOWING  QUESTIONS  AS  SOON  AS  YOU  HAVE 
COMPLETED  THE  SESSION.  YOUR  RESPONSES  SHOULD  FOCUS  ONLY  ON  THE 
WORK  THAT  YOU  HAVE  JUST  COMPLETED  IN  THE  LAST  CONTROL  PERIOD. 

ALL  CONTROLLERS  EXPERIENCE  A  WIDE  VARIETY  OF  ACTIVITY  AND  RESULTANT 
WORKLOAD  DURING  THEIR  CAREERS.  IT  DOES  NOT  DETRACT  FROM  YOUR 
PROFESSIONALISM  IF  FOR  A  GIVEN  PERIOD  YOU  REPORT  VERY  HIGH  OR  VERY 
LOW  WORKLOAD.  ON  ALL  THE  QUESTIONS  WHICH  FOLLOW  FEEL  FREE  TO  USE 
THE  ENTIRE  NUMERICAL  SCALE  FOR  EACH  ANSWER.  BE  AS  HONEST  AND  AS 
ACCURATE  AS  YOU  CAN. 


9 
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1.  Rate  your  performance  controlling  traffic  during  the  past  session.  Circle  the  number 
which  best  describes  how  well  you  think  you  did. 

123456789  10 

average  excellent 


2.  What  fraction  of  the  time  were  you  busy  during  the  period  you  were  controlling? 


1  2  3  4  5  6  7  8 

9 

10 

seldom 

fully 

had  much 

occupied  at 

to  do 

all  times 

How  much  did  you  have  to  think  during  this  period? 

1  2  3  4  5  6  7  8 

9 

10 

minimal 

a  great  deal 

thinking  & 

of  thinking  & 

concentration 

concentration 

4.  Rate  the  degree  to  which  you  found  this  control  period  stressful. 


123456789 

low 


10 

high 


5.  I  am  feeling  tired. 


1  2  3  4  5  6  7  8 

strongly 
disagree 


9  10 

strongly 

agree 


6.  Briefly  describe  your  strategy  for  working  traffic  during  this  control  period. 


7.  If  you  have  a  choice  of  separating  aircraft  vertically  (by  altitude)  or  horizontally,  which 
do  you  prefer  to  do  and  why? 
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8.  How  helpful  were  the  paper  flight  strips  to  you  in  making  control  decisions  for  aircraft 
during  this  past  session? 

123456789  10 

not  at  all  extremely 


9.  Briefly  explain  why  the  strips  were  or  were  not  helpful.  Compare  your  performance 
during  this  session  to  your  performance  in  other  sessions  when  the  strips  were  used  differently. 


10.  To  what  extent  were  tb  paper  flight  strips  useful  for  helping  you  remember  aircraft 
information  when  the  display  was  blocked  from  view? 

123456789  10 

not  at  all  extremely 

12.  Briefly  explain  why  the  strips  were  or  were  not  helpful.  Describe  anything  else  that  you 
feel  might  have  been  helpful  in  getting  you  to  remember  this  information  more  quickly  or  more 
accurately  than  you  did. 


13.  Describe  the  strategies  you  used  to  help  you  remember  aircraft  information. 


Is  there  anything  else  that  happened  this  past  session  which  you  feel  might  help  us  understand 
the  results?  Any  comments  you  have  at  this  point  would  be  very  welcome. 
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